WILLINGNESS TO PAY AT PR19

Transcription

1 WILLINGNESS TO PAY AT PR19 A Note by PJM and Accent: 13 November INTRODUCTION 1.1. Background Following the completion of the 2014 price review, industry participants are now taking stock and looking ahead to PR19. Ofwat recently published a wide-ranging review of PR14 (Ofwat, 2015a) and has also more recently issued a consultation document, "Towards Water 2020" (Ofwat, 2015b). This included a series of policy notes covering various aspects of its methodology, the most pertinent of which from the perspective of Accent and PJM concerned "Customer Engagement and Outcomes" (Ofwat, 2015c), which contained some interesting views on the collection and use of WTP evidence. Additionally, there has been an UKWIR review of customer engagement practices, conducted by Blue Marble research (UKWIR, 2014), and several companies, including Northumbrian Water, Yorkshire Water, Southern Water and Wessex Water, have published their own discussion documents relating to customer engagement and outcomes (Northumbian Water, 2015; Yorkshire Water, 2015; Southern Water,2015). It is clear from review of these publications, as well as from our own conversations with a number of water companies post-pr14, that there are some questions with the way that WTP evidence was collected and used at PR14, and that the approach to be taken at PR19 may need to be re-thought Objectives The purpose of this note is to capture the various critiques made of the PR14 WTP approach, including some of our own, and to set out our views on how the approach should evolve looking forward. We hope that this note serves to prime a discussion with companies on how to develop a forward-looking approach to the collection and use of WTP evidence for PR19 business planning Structure of Note The remainder of the note is structured as follows. Section 2 reviews the key features of the approach taken at PR14 to the collection and use of WTP evidence; documenting the various critiques that have been made of the approach recently; Section 3 sets out our views on how the approach should evolve going forward, including a discussion of the key issues that we believe must be addressed; Section 4 draws some conclusions. 2. REVIEW OF PR14 APPROACH 2.1. Key Features of PR14 WTP Surveys The principal approach taken at PR14 to collect WTP evidence was based around the use of multiattribute choice experiments. This involved presenting respondents with sequences of choices PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 1 of 9

2 between alternative packages of changes to their water service. Importantly, these choices included trade-offs between service changes and bill levels and so it was possible, via analysis of respondents' choices, to infer their willingness to pay. The WTP measures were obtained individually for incremental changes to a wide range of service measures. Although the actual survey instrument used varied from company to company, the methodologies were structurally similar, principally due to the fact that a prior UKWIR study had provided detailed guidance on how such surveys should be conducted (UKWIR, 2011). A great deal of thought and consultation - including with expert practitioners, asset planners and industry regulators and stakeholders - had gone into the design of the UKWIR guidelines and the resulting survey instruments represented a clear advance on what had come before at PR09 in many respects. One of the key strengths of the PR14 approach was that it could generate WTP estimates for a wide range of potential service improvements all within the context of an overall package of improvements. It could thereby allow for performance commitments (PCs) to be optimised using individual WTP numbers by attribute, whilst avoiding the known adding-up bias whereby WTP values for a package tend to be less than the sum of their parts, where these parts are individually estimated. The PR14 WTP approach also provided a mechanism for linking separate WTP surveys together, (socalled 'stage 2 surveys'), in such a way that additional attributes could be assigned WTP values that were also consistent with the package of improvements valued in the 'stage 1' WTP survey. For example, a stage 1 survey could value a reduction in the number of interruptions generally, characterised by their average duration, and the stage 2 survey could then drill into this value to test the relative values of different durations of interruption. This approach potentially allowed for many more service measures to be valued consistently than would be possible via a single survey instrument. The PR14 approach was, and still is, consistent with academic/policy best practice guidelines, and furthermore, results tended to perform well against all the usual measures of validity, including tests of understanding, effort and concentration, and based on the conformance of results with expectation across a number of areas. Overall, the PR14 approach can be judged as being successful on many counts, and it is reasonable to argue that there have been no other proposals put forward to date that can be judged better, all things considered. In particular, it is possible to dismiss simple market research techniques such as importance scales, and priority scores/rankings fairly easily as replacements to the choice-based WTP approach on account of their failing to generate WTP evidence of any kind that can be used within cost-benefit analyses (CBA). Nonetheless, despite the achievements of the PR14 approach, there are/were weaknesses, and a number of these have been brought to the fore in recent industry reviews and discussion papers. We document and discuss these critiques in Section 2.3, but first 2.2 discusses how the results were applied at PR14 and highlights some issues therein. PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 2 of 9

3 2.2. How WTP Evidence Was Used at PR14 The PR14 WTP approach was designed to collect evidence to support the setting of PCs via CBA. There is a clear underlying economic model into which WTP evidence fits such that, in conjunction with cost data, PCs can be optimised for economic efficiency, and it was to cohere with this model that the PR14 WTP surveys were designed. However, this wasn't quite how it turned out at PR14. Firstly, Ofwat intervened at the draft determination stage to revise PCs to upper quartile (UQ) levels for five outcome measures, disregarding the costs and benefits of doing so. These measures represented some of the key outcomes that customers expect companies to focus on. The principal rationale behind Ofwat's intervention was that the PR14 WTP surveys conducted by companies were not always robust, due to the following reasons. 1) There was an extremely wide disparity between WTP values from different companies. 2) They may depend on other circumstances beyond the scope of the survey, such as income levels and other competing demands on incomes." 3) WTP values are sensitive to framing, and in particular, to information on how well the company is, or will be, performing on a certain measure; and this information was not shown to respondents. 4) Customers can find it hard to work out what they are willing to pay to reduce the probability of a bad, but unlikely, event from occurring. As a consequence, Ofwat deemed that the WTP data were not robust for some measures, and that it was hence justifiable to replace PCs which were set via CBA using these numbers to PCs based on UQ levels. Notwithstanding the validity or otherwise of Ofwat's arguments, which is discussed in Section 2.3, it is almost certainly the case that, as a consequence of its intervention, PCs will no longer have been economically efficient when increased to the UQ level. In other words, there will almost certainly be cases where it costs more to achieve the PCs specified in the Final Determinations than customers would be willing to pay for. The approach also introduces an inconsistency in Ofwat's approach to setting PCs across outcome measures. Some outcome PCs were set to UQ levels; others were set via CBA, and these two approaches do not easily co-exist. Aside from PCs, WTP values were also used at PR14 to set outcome delivery incentive (ODI) rates. ODIs reward companies for exceeding PCs, via allowing higher bills, and punish them, via lower bills, for falling short against PCs. They were put in place by Ofwat to give companies the appropriate incentives to meet PCs, rather than save the money, and to exceed them where WTP outweighed the cost of doing so. WTP evidence was hence needed, and used where possible, to set ODI rates. However, the PR14 WTP approach was not designed with this in mind and results are not necessarily valid for this use PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 3 of 9

4 due to the fact that WTP is a measure derived from choices in their correct context, not for the object of the valuation per se. Changing the context could change the valuation. On the other hand, the achievement of economic efficiency with respect to the actual outcome level achieved demands that the same WTP values should be used both for CBA setting of PCs, and for ODIs. This is because it is the same object being valued in each case. There is therefore an issue to be considered in respect of the future WTP approach about how to measure WTP when it is to be used in both contexts. For various reasons, incentives may not necessarily be economically efficient under the approach taken by Ofwat/companies to the setting of ODI rates. For example, in cases were PCs were set to the UQ level rather than via CBA, economically efficient incentives would encourage companies to under-deliver against their PCs. Ofwat's incentives were designed, reasonably in the circumstances, to ensure that they would be met. Additionally, ODIs include a cap and collar, plus deadbands where no penalty or reward applies. These are generally reasonable, but do tend to blunt the incentives in certain circumstances. More significantly, many ODIs can be somewhat baffling from a distributional perspective to anyone outside of the industry. Why, for example, should a customer with perfect service be compensated ex-post for someone else being flooded with sewage? Even if there is altruistic willingness to pay for reduced sewer flooding at other people's properties, it would seem strange that non-affected customers should be compensated rather than those harmed by the flooding. This type of consideration suggests that it may be beneficial to consider enhanced compensation payments in lieu of general bill reductions to account for any penalties owed by the company due to service shortcomings, as in fact at least one company (Welsh Water) did at PR Critiques of PR14 WTP approach The following list summarises the main critiques put forward against the PR14 WTP approach, from Ofwat's Towards Water 2020 discussion paper, Blue Marble's UKWIR report, which included a survey of companies and stakeholders, and recent Water 2020 discussion papers by Northumbrian, Yorkshire and Southern Water on related topics. Too complex and/or abstract Choices don t relate to the real world Consumers can t value small risk changes Unclear how answers feed into business plans/a black box Measures don t take account of comparative performance WTP numbers are too variable across companies to be valid Measures aren t consistent with how they are applied in ODIs Complex/abstract nature of choices The majority of criticisms made of the PR14 WTP approach have related in some way to the complexity of the choices facing customers, and/or their abstract nature. A particular issue with the PR14 WTP approach was due to the fact that many service measures were characterised as the risks PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 4 of 9

5 of a service failure of some kind, and those risks were almost always very small. In addition, survey instruments tended to include a large number of attributes (often 10-12), many of quite an abstract nature to respondents, and furthermore, the overall choice context facing respondents would have been unfamiliar in these surveys. Although PR14 WTP studies tended to perform reasonably well against all usual tests of validity put forward in the academic literature, including tests of understanding, effort and concentration, and based on the conformance of results with expectation across a number of areas, it is still a fair criticism that the study was complex and somewhat abstract. It is perhaps the key challenge for the future evolution of WTP approach to try and capture the same detailed WTP data, but via an instrument that engages customers on a simpler level. The black box issue Related to the issue of complexity is the problem that many stakeholders did not understand how the WTP results were derived from the choice data, and how they were used within the business plan process. CBA and the use of econometric modelling were inevitably unfamiliar topics to many internal and external stakeholders and, from our own experience, where there was no, or little, engagement between the WTP researchers and the Customer Challenge Groups (CCG), the analytical approach tended not to be trusted so much as when there was early and frequent discussions between us, the researchers, and the CCGs. For PR19, it is clear to us that it would be helpful to make these sorts of discussions an integral part of the CCG process. Measures don t take account of comparative performance As mentioned in Section 2, a key critique of Ofwat s, which led them to abandon the use of WTP values and CBA as the basis of setting some of the key performance commitments, was that WTP values tend to be sensitive to how questions are framed, and in particular, are likely to depend on comparative performance information which was not shown to respondents. This issue was never discussed in the UKWIR (2011) guidelines on carrying out WTP surveys, nor was it ever mentioned by Ofwat, to our knowledge, prior to the Draft Determination stage of the price review. Nonetheless, provided that comparable data are available to the WTP researchers at PR19, it should be straightforward to include this within the survey instruments, and thereby resolve the problem headon. It is not clear, in advance, what effect, if any, this change might make to the results. WTP numbers are too variable across companies to be valid A further critique made by Ofwat, and also by other stakeholders, is that the WTP values varied dramatically across companies for notionally the same outcome improvement. For example, as Ofwat reported, willingness to pay to avoid sewer flooding ranged from 25,540 to 434,319 per property across companies. There are good and valid reasons why WTP may vary substantially from company to company, namely differences in customers preferences from region to region. However, the size of the differences in WTP suggest that there may be issues relating to the survey design, perhaps the lack of contextual information being shown to respondents about comparative performance, that are causing them. The issue suggests a further reason why it may be helpful to include such information within WTP survey instruments, although it is to be expected that substantial differences across companies may still remain. PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 5 of 9

6 Measures need to be triangulated with/supported by other evidence A number of comments have been made, including by Ofwat, that the WTP measures should be supplemented by additional evidence on customers preferences. The high degree of variability across companies in WTP for similar outcome improvements also suggests that it would be worth trying to obtain alternatively sourced WTP measures, either from revealed preferences/customer behaviour where possible, or from distinctly designed survey instruments, to triangulate the results, and thereby have more assurance as to their validity. Measures aren t consistent with how they are applied in ODIs In a Water 2020 paper published by Northumbrian Water (NW), the company discusses a number of issues raised by Ofwat s approach to the use of the WTP values at PR14. In particular, NW noted that the use of the values to set ODI rates was not consistent with the contextual framework within which they were derived, which was based on the setting of performance commitments for AMP6, even in cases where the outcomes being incentivised were the same as those included in the WTP surveys, which was not always the case. In general, WTP values are sensitive to the framing of the question, and it is not necessarily the case that the resulting values would be the same when obtained in the context of setting ODI rates as they would be when obtained in the context of setting performance commitments. However, economic efficiency requires that the ODI rate is based on the same WTP value as was used in the CBA to set the performance commitments. For PR19, it will be important to consider how best to derive WTP values that are valid for both types of application. 3. WHERE NEXT FOR PR19 WTP? The previous section has highlighted a number of directions along which the WTP approach for PR19 should try and evolve. These include the following. Simplify the instrument and make it less abstract Have WTP researchers and CCGs engage early to discuss the methodology Include comparative performance information within surveys Conduct distinct but comparable research to test, validate and triangulate with the main WTP results Aim to estimate WTP values that are valid both for setting performance commitments, via CBA, and to use as the basis for setting ODI rates. All of these directions are, in our view, worth pursuing. In the remainder of this section,we share our initial thoughts on how one might go about simplifying the instrument, and making it less abstract, without abandoning any of the principles that made the UKWIR (2011) WTP guidance. PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 6 of 9

7 3.1. A simpler/less abstract survey instrument Perhaps the key difficulty with the PR14 survey instrument, as regards complexity, was the fact that it required respondents to process and evaluate the benefits of very small risk reductions. It would make the survey much simpler if dealing with small risk changes could be avoided altogether. There were two related reasons within PR14 WTP surveys for showing small risk changes to respondents, which could, in principle, be separated. One reason for showing them was to enable the relative value of each service measure improvement to be estimated; the second reason was to capture the entirety of a package of service improvements covering all, or at least a large number, of service measures, in order to measure customers WTP for that package. One way of avoiding the need to ask customers about small risk changes is to impose the principle of rational choice under risk, also known as Expected Utility Theory (Von-Neumann and Morgenstern, 1953). Under this principle, the utility, or value, that a customer gets from a small reduction in the risk of a service failure is equal to the change in probability multiplied by the loss of utility associated with the service failure itself. For example, a reduction from 2 in 10 to 1 in 10 in the chance of a supply interruption would be valued as (2/10-1/10) times the value of an avoided supply interruption. This makes the valuation potentially much simpler because one only needs to know the value of an avoided supply interruption to value any reduction in the risk of a supply interruption. The advantage of imposing this principle in the case of water company WTP surveys is that one can focus on the relative aversion to different types of service failure directly, and then simply scale these values by the extent of the service improvement, measured by the change in risk per customer, in order to measure the relative value of each of the candidate service improvements. In other words, there would be no need to ask respondents to consider small risk changes in order to understand the relative values of different types of service improvement. The approach forces rational choice under risk, which can be seen as an advantage or a disadvantage depending on one s perspective. It is an advantage in the sense that rational choice is an appealing principle to adhere to; it is a disadvantage in the sense that there is clear behavioural evidence that people do not always choose rationally under risk. (A great deal more could be written here about the pros and cons of Expected Utility Theory. For brevity, however, that is left for future discussions.) There would still be the need to estimate WTP for a package of service improvements under this approach in order to determine the overall ambition of the company s business plan. Obtaining estimates of WTP for a package of measures enables the apportioning of value to individual service measures via the approach described above. There would be an additional need to test preferences with respect to ODI rates, as this would also generate evidence on customers valuations. As discussed previously, the resulting values may differ from WTP values for packages of improvements due to the different contexts in which valuation took place. The choices that customers would need to make in order to establish their WTP for packages of service improvements could be between, say, four high level options for the future of their water PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 7 of 9

8 service. One option could involve a bill reduction, one a constant bill, one a small increase to the bill, and one a more substantial increase to the bill. The options could be described in much simpler, and less abstract terms, than in the past, and could also make use of comparative performance information. This would require some careful work by the company and its researchers, but also, and importantly, the company and its researchers could consult with stakeholders at an early stage to ensure that there was widespread buy-in concerning how the packages are represented to customers in relation to how the company intends to interpret the results. The choice questions would need to include bill impacts that varied across respondents according to an experimental design in order to estimate the distribution of WTP across the customer base. Options would not need to be given accurate costs since what matters for the purposes of WTP estimation is how many people would choose each of the options at different bill levels. Provided that there was sufficient variation in the costs shown for each of the options, and a spread of choices over the options shown, estimates of WTP for moving between each of the package options would be obtained. With regard to measuring customer values consistent with setting ODI rates efficiently, there would potentially be different types of questions needed depending on how the ODI was to be structured. For example, if companies decided to link ODIs to enhanced compensation payments then the relevant survey questions would focus on the levels of payments that would be thought to adequately compensate customers in cases where they experienced a service failure. On the other hand, different questions would be warranted, and different values may emerge, in cases where ODI penalties and rewards were to be pooled into general bill reductions or increases. In terms of sequencing the research, we would anticipate that there would be two phases of WTP research. The first would take place at an early stage of the programme, at which point it would not be necessary for any of the options shown to correspond to the company s ultimate business plan. A follow-up stage would then be conducted once business plans were more firmly in place, to refine the options put to customers and thereafter to refine the business plans if and where necessary. Separate research would likely to be needed on ODI structures and rates. Acceptability testing research would be the final stage of the process. The above has set out a very high level sketch of how we anticipate the WTP research might evolve. There are clearly gaps that would need filling in to develop a fully-fledged approach. We put this forward to companies to initiate discussions on the high level ideas herein. 4. CONCLUSIONS As at PR14, a core customer engagement requirement for PR19 business planning will be the need to measure customers WTP for service improvements. Whilst there have been many recent criticisms of the PR14 WTP approach, it can be judged objectively as having been successful on many counts. However, it is timely to consider how the WTP approach should evolve going forward. The core areas for reform, as identified in this note, are the following. Simplify the instrument, and make it less abstract Have WTP researchers and CCGs engage early to discuss the methodology PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 8 of 9

9 Include comparative performance information within surveys Conduct distinct but comparable research to test, validate and triangulate with the main WTP results. Aim to estimate WTP values that are valid both for setting performance commitments, via CBA, and to use as the basis for setting ODI rates. We have set out some high level views in this note on how the approach might be simplified and made less abstract. We are distributing this note to companies to initiate discussions on these ideas, and on the next steps, more generally, in the journey towards PR REFERENCES UKWIR (2011) Carrying Out Willingness to Pay Surveys, UKWIR Report, By NERA and Accent UKWIR (2014) Post PR14 Customer Engagement, Communications and Education, UKWIR Report, By Blue Marble Research Northumbrian Water (2015) Improving the Outcome and Delivery Incentive Regime, A Water 2020 Discussion Paper Ofwat (2015a) Reflections on the price review -learning from PR14, Ofwat Report Ofwat (2015b) Towards Water 2020 meeting the challenges for water and wastewater services in England and Wales, Ofwat Discussion Paper Ofwat (2015c) Towards Water 2020 policy issues: customer engagement and outcomes, Ofwat Discussion Paper Southern Water (2015) Customer Engagement: Lessons and Opportunities, A Water 2020 Discussion Paper Von-Neumann, J. and Morgenstern, O. (1953) Theory of Games and Economic Behaviour, 3 rd Ed., Princeton, NJ Yorkshire Water (2015) Customer Engagement, Insight and Service, A Water 2020 Discussion Paper PJM Economics/Accent Accent-PJM_PR19 WTP_v3.docxPJM Page 9 of 9