Revolution, Evolution, or Status Quo? Guidelines for Efficiency Measurement in Health Care

Transcription

1 Revolution, Evolution, or Status Quo? Guidelines for Efficiency Measurement in Health Care Professor Bruce Hollingsworth Director, Monash University

2 Efficiency Measurement Applied academic research into efficiency has grown into a thriving industry Hollingsworth and Street, Health Economics, 2006; 15(10):

3 The problems Thriving supply side Published literature of 400+ journal papers 250 about hospitals Not such a healthy demand for such work (policy makers, managers) Why? 3

4 Hypotheses Why is transfer of knowledge not happening now? Not used because not useful? Is there a need to make the evidence being produced more useful in leading to changing practice? 4

5 A way forward One way forward in terms of making work in this area more useful may be a set of guidelines - with regard to how to conduct efficiency analysis in this area Are guidelines really needed? 5

6 A way forward (2) Can draw upon the example of the recent history with respect to economic evaluation in health care Efficiency measurement is closely related to economic evaluation in many respects So a similar approach to guideline development may be justified 6

7 A way forward (3) This process is seen as a starting point, the evaluation guidelines took a number of years to evolve, and continue to do so The evolution of guidelines for the conduct of efficiency measurement in health care may take just as long Here the beginnings of this process are suggested 7

8 Background Growing pressure internationally to improve the performance of health services As a consequence - develop metrics and evaluation techniques with a scientific underpinning Summaries of the literature in the area of efficiency measurement in health care have reported on the diversity of data used, techniques, models, and areas of analysis 8

9 Background (2) Healthy supply of studies, unhealthy translation into service delivery/policy change May be that studies are not comparable, or applicable in settings outside specific remit Previously suggested use and usefulness criteria for efficiency measurement studies Process may need to be developed more formally into actual guidelines, as in evaluation 9

10 Economic Evaluation Efficiency measurement and economic evaluation can be viewed as (near) equivalents Both seek to estimate some form, or part of, economic efficiency Efficiency analysis may never be used as often in policy terms as evaluation of medical interventions which involve a clear choice 10

11 Economic Evaluation Inclusion of allocative measures may help make efficiency studies more tractable in policy and service delivery terms More easily recognised as to how comparable production units or techniques are when specified in monetary units 11

12 Guideline Development In economic evaluation - two alternative pathways One evolved over a number of years (Drummond et al) Evolution The other in a one off hit (Gold et al panel) Revolution Nice contrast in process terms 12

13 A Panel the Revolution Gold panel - 13 individuals, convened 1993, book published 1996 Remit (from US Public Health Service): assessing state of the science; identifying methodological inconsistencies and fragilities in the technique; fostering consensus with respect to standardizing conduct of studies; and proposing steps that can be taken to address remaining issues and uncertainties in the methodology. 13

14 Terminology Issue highlighted by Gold is that the term cost-effectiveness is not used in a precise manner the same is true of the term efficiency Different disciplines having different meanings, based on differing concepts Producers of efficiency measures may mean something different by efficiency than users of such measures 14

15 Comparability If aiming at comparability of studies, would we ever get as far as the economic evaluation concept of league tables? What would the usefulness of such comparisons actually be? If comparability is an aim do we need a reference case? This would involve agreement on a standard set of methodological practices an analyst should seek to follow 15

16 Comparability (2) Could involve a minimum set of standards Not intended as suggesting that this is all that is ever done Advances in economic evaluation techniques demonstrate a set of common standards does not hold back techniques - In fact they may act as a spur to further such advances if they highlight deficiencies in the quality of information available 16

17 Evolution The other approach to guideline development is more evolutionary, over a longer period, developed in the literature, i.e. a little more akin to the approach of Drummond et al (2005) Why have a common reporting framework? To increase transparency; to allow comparison between studies; and to improve the quality of studies (if researchers know what is expected of them) 17

18 Use Guidelines can have different uses for different users Drummond et al establish that they can be for critical appraisal of studies, or to improve the quality of studies by those undertaking them Several sets of reporting guidelines now exist for evaluations, eg NICE, PBS Evaluation is now seen as a key decision making aid 18

19 Use and usefulness Development and use of guidelines may lead to the methods becoming more useful Potentially actually being used more by decision makers/service delivery The development and acceptance of rigorous guidelines in CEA have led to use in decision making in health care systems Guidelines have improved the quality of work undertaken, reassuring users that the work is of value 19

20 Usefulness So what is the purpose of a system of guidelines? There are guidelines as to how to formulate, implement and disseminate guidelines (for example Grimshaw et al, 2005) Clinical guidelines developed to bridge the gap between research findings and clinical practice, given concerns that variations in clinical practice may impact upon patient outcomes 20

21 Outside economics Reviews of guidelines have shown them to be an extremely effective means of changing service delivery and so ultimately health outcomes Evidence based guidelines are seen as an essential means of ensuring the delivery of high quality health services Adherence to clinical guidelines may increase quality and reduce costs by up to 25 per cent (Rosenbrand et al, 2008) 21

22 Transferability Information is generally applicable in a local setting An organisation/process efficient in one setting is not necessarily so in another Lots of reasons different prices, different practices, different objectives, differences in incentives, population demographics, access issues, etc. 22

23 Transferability (2) Could efficiency studies be adapted by modelling, the way economic evaluations are, to different settings? Would studies ever be comparable enough to go in a league table format, and in what sense would league tables of efficiency measures be used to compare different specialties in hospitals, or between hospitals for example? 23

24 What might guidelines look like? If it was agreed that some form of guidelines would be useful, what might they look like? There are many forms guidelines can take There is a clear place for detailed methodological guidelines, for the techniques used e.g. DEA/SFA - lack of consensus on the specification, testing and selecting of models 24

25 What might guidelines look like? (2) A number of good texts on techniques Limited work on how to practically undertake a study that is sound, and useful to end users Guidelines can suggest the process to follow in methodological terms List could include: moving from theory, to model specification, to sensitivity analysis, to data testing, through to modelling and significance testing etc. 25

26 Use and usefulness criteria Is there a way of undertaking research in this area which would make the supply of studies more effective? In other words are there specific criteria which would make efficiency measurement more user friendly? Establish some initial non-exhaustive criteria as a starting point, in both macro and micro terms 26

27 Use and usefulness criteria Suppliers Macro issues Applied research needs to be placed in a policy context get end users involved early on, helps ownership feedback results at various stages, and to different levels of users Hospital managers may have concerns about health authorities using efficiency measures as big sticks interested in detailed information, health authority interested in big picture and comparisons what value are you adding to the way efficiency is measured already? Have you given your end users the information you set out to? surveying them disseminate your results limitations of efficiency measures, a useful policy tool, not the useful policy tool 27

28 Use and usefulness criteria Suppliers Micro Issues Are you asking the right questions? What is your underlying economic theory of production (or cost)? Is your model specified correctly? Have you undertaken extensive sensitivity analysis? Are your data good enough to answer the questions? Have you any data on quality? Is your sample inclusive, are you comparing like with like? Sample size Which techniques will you use? Panel data? Do you need to generate confidence intervals? 28

29 Use and usefulness criteria - Demanders A check-list for assessing if an efficiency analysis should be made use of This is a starting point, similar in concept to the Drummond et al list Suppliers of efficiency studies may also wish to take note of these points The two assessment questions asked by Drummond are also pertinent here: is the methodology appropriate and are the results valid; and if the answer to this is yes: do the results apply in my setting? As Drummond et al acknowledge, it is unlikely every study can fulfil every criteria, but criteria are useful as screening devices to identify strengths and weaknesses of studies 29

30 A 10 point checklist for assessing efficiency measurement studies 1. Is the question well defined, and answerable? - inputs and outputs clear? - is there a particular viewpoint stated (whose objectives?), decision making context? 2. Is a comprehensive description of the sample given? - relevant comparator units excluded? - sample strictly comparable, outliers? 3. Are the quality and quantity output data clear and comprehensive? - who collected data, why? - case mix adjusted? - are quality data useful? 4. Are all the relevant inputs and outputs included? - range wide enough? - do they cover all relevant viewpoints (e.g. policy makers, managers) - physical quantities of inputs as well as costs 30

31 A 10 point checklist for assessing efficiency measurement studies 5. Are inputs and outputs measured accurately in appropriate units? - resources used accounted for? - omitted data? Justification? - special circumstances, e.g. joint use of staff? Handled appropriately? 6. Were inputs and outputs (or objectives) valued (or weighted) correctly? - sources of all values identified? E.g. market prices for inputs, case mix weights? - value of outputs appropriate? Weights placed upon relationship between quantities (and qualities) of outputs? 7. Were analyses over time undertaken? - values adjusted to present value? - specific techniques justified, e.g. RE/FE models used, scale 31

32 A 10 point checklist for assessing efficiency measurement studies 8. Do techniques add incremental value? - for example is DEA used? Or SFA? - are techniques justified, e.g. what incremental value do they add? 9. Was allowance made for uncertainty? - statistical analyses? - sensitivity analyses - which dimensions are tested? - results sensitive to the statistical/sensitivity analysis? 10. Did the presentation/discussion of results include all issues of concern to users? - conclusions based on an overall measure, or individual comparisons? - results compared with others? Generalisability? - other important factors, e.g. ethical, or access, or equity? - implementation, e.g. feasibility of adopting efficiency changes, given operational constraints, can freed resources be redeployed to more efficient programmes? 32

33 User involvement End users of analysis may benefit from being involved in guideline development translational research Users are more able to make use of results produced Adherence to an agreed process might help the users of efficiency analysis have confidence in the analysis - that it is the result of a consistent and scientifically valid process 33

34 Or just guidance? Could offer guidance, in terms of advice and support NICE offers a set of templates, tutorials and implementation tools Guidance in the area of efficiency measures may take the form of general how to guides for those undertaking or interpreting results, guidance as to how to present results in a useful manner 34

35 Guidance (2) Given such guidance there may be a choice of pathways to obtaining optimal practice the same as there are several treatment patterns available in some clinical guidelines, all of which may lead to a similar health outcome Guidance could be made available via a clearing house, like the US National Guideline Clearing House (NGC) 35

36 A process for development A formal panel could be established, with the aim of defining across the board best practice guidelines for undertaking, disseminating and interpreting efficiency measurement in health care But given the evolution of the process in this area, re-visiting issues on a regular basis would be important This would mean a more evolutionary process 36

37 A process for development (2) Evolution - but could feature a one off panel meeting and establishing of baselines Useful revisions could be made by making use of the internet Either in the more formal (and more prescriptive) manner of NICE, or the less formal making available of information of the NGC House 37

38 A process for development (3) All of this would require commitment of resources, especially in set up terms The Gold panel met 11 times over a 2 ½ year period What about keeping information up to date and current, i.e. useful These are not insignificant considerations, and would be helped by a major body or group of individuals taking ownership of such a process 38

39 Conclusions Pressure on Governments to make efficient use of scarce resources in the hospital (and other health care) sectors is intense There is an excellent supply side in the area of provision of potentially useful information This is not matched by demand To help match up demand and supply, guidelines on how to undertake efficiency measurement studies may help 39

40 Conclusions (2) Guidelines can help in many ways: helping those commissioning such work to know exactly what to ask for; helping those who are commissioned to undertake such research to undertake cutting edge research, and; to produce results which are actually of some use to policy makers and those involved in service delivery 40

41 Conclusions (3) A decision has to be made as to the efficiency of this exercise, in terms of inputs to the development process related to the usefulness of any guidelines produced One aim here is to raise issues for discussion and suggest some possible ways forward, some of which in other related areas have proved incredibly useful, and have aided in the decision making process 41

42 Conclusions (4) Perhaps a revolution is needed in terms of the establishment of a panel to formulate guidelines in this area This could be followed by an evolutionary process to take the guidelines forward, given the state of play will change over time In the end judgements have to be made and efficiency measures have to prove themselves as a useful aid to decision making 42