THE ISPOR MCDA TASK FORCE: HOW BEST TO USE IT IN HEALTH CARE DECISION MAKING. Kevin Marsh, Maarten IJzerman, Praveen Thokala and Nancy Devlin

Transcription

1 THE ISPOR MCDA TASK FORCE: HOW BEST TO USE IT IN HEALTH CARE DECISION MAKING Kevin Marsh, Maarten IJzerman, Praveen Thokala and Nancy Devlin May 19, 2015 Kevin Marsh Evidera 1

2 To share preliminary recommendations of the MCDA emerging good practice task force To develop good practice recommendations for researchers and decision makers on the application of MCDA in health care decision making* The task force will: Provide a common definition for MCDA in health care decision making Develop emerging good practices for conducting MCDA to aid health care decision making *All type of decisions, e.g. pipeline investment, authorization, reimbursement / coverage, prescription 2

3 Co-Chairs Maarten J. IJzerman, University of Twente, Netherlands Kevin Marsh, Evidera, London Nancy Devlin, Office of Health Economics, London Praveen Thokala, University of Sheffield, Sheffield Leadership group Meindert Boysen, National Institute for Health and Clinical Excellence Zoltan Kalo, Eotvos Lorand University, Budapest Thomas Lonngren, NDA group AB, UK and Sweden Filip Mussen, Jansen Pharmaceutical, Antwerp Stuart Peacock, British Columbia Cancer Agency, Vancouver, Canada John Watkins, Premera Blue Cross, USA 3

4 Paper 1: Introduction to MCDA The definition of MCDA An overview of the decisions for which MCDA is/could be used An introduction to the types of MCDA and common steps Paper 2: Emerging good practice guidelines for how to implement MCDA: Selecting and implementing methods at each of the MCDA steps Scientific factors, including a typology of scoring and weighting techniques Practical factors, resources and skills MCDA checklist 4

5 Maarten IJzerman: Introduction to the MCDA checklist Praveen Thokala: More detailed good practice recommendations: (i) selecting criteria, (ii) choosing scoring / weighting methods Nancy Devlin: Some issues for discussion Maarten IJzerman University of Twente 5

6 Emerging good practices Taskforces identify new and emerging methodologies for use in outcomes research Checklists a a result of Taskforces: Educational, i.e. instruction of people new to the field Improving the design and standardize reporting Specific guidance on design issues Widening the use of these methods in outcomes research Examples of other checklists Conjoint analysis ( a checklist ) Bridges et al, Value in Health 2011) Health economic evaluation (CHEERS) Husereau et al, Value in Health 2013 Dynamic simulation modeling (SIMULATE) Marshall et al, Value in Health 2015 It is doubtful if an identification of the best MCDA method in general can be performed (De Montis et al, 2005) It is impossible to characterize all the DMS; there might exist as many DMS as there are decisions (Guitouni and Martel,1998) All methods have their assumptions and hypotheses, on which is based all its theoretical and axiomatic development - these are the frontiers beyond which the methods cannot be used (Guitouni and Martel, 1998) 6

7 MCDA Checklist ought to produce guidance on Understanding critical issues in conducting MCDA studies Designing MCDA studies Reading and interpreting published MCDA studies The checklist does not intend to guide nor prescribe specific choices in doing MCDA, such as the selection of a weighting approach Feedback requested! m.j.ijzerman@utwente.nl MCDA step Recommendation Definition Develop a clear description of decision problem including the objectives, the 1. Decision stakeholders, and alternatives, and whether the objective is to value or rank problem alternatives. Validation Validate the decision problem with decision makers and clinical experts Develop Report the sources used to identify criteria, the long list of criteria identified, 2. Selecting and the rationale for exluding criteria. criteria Report Report the final criteria list, including definitions and measurement scales. Validation Validate the criteria with stakeholders and against MCDA requirements (complete, non-redundant, non-overlapping, preferentially independent). 3. Measuring Methods Report and justify the sources uses to measure performance performance Reporting Report the performance matrix, showing performance of alternatives against criteria MCDA model Report the MCDA model (value measurement, outranking, reference), and 4. Scoring and justify in reference to the decision problem and stakeholder preferences. weighting Scoring / Justify the methods used for scoring and weighting with reference to the weighting objectives of the analysis and the stakeholders involved in the analysis. methods Reporting Report the values of scores and weights Validation Validate the meaning of scores and weights with stakeholders 5. Aggregation Reporting Report the aggregation function used in the analysis Validation Validate with stakeholders that the aggregate scores reflect how they expected their scores and weights to be used 6. Dealing with Report Report and justify variation (ranges, distribution) in parameter inputs, uncertainty including sources, and list all assumptions made (including structural assumption). Characterizing Consider the effects on the results of parameter uncertainty and uncertainty assumptions. Uncertainty At a minimum deterministic sensitivity analysis should be performed. Justify methods the method adopted to explore uncertainty with reference to the differences between techniques Characterizing Consider the effects on results of variation in scores and weights and heterogeneity performance measures between sub-groups. 7. Robustness The findings should be interpreted in light of the results of the validation Interpretation undertaken and the results of the analysis of the impacts of uncertainty and heterogeneity. Meaning The finding should be interpreted in light of the meaning of the particular scores and weights employed. 7 items with 21 sub-items: 1. Decision problem 2. Selecting and structuring criteria 3. Measuring performance 4. Scoring and weighting 5. Aggregation: analysis and report 6. Dealing with uncertainty 7. Interpretation 7

8 MCDA step Recommendation Definition Develop a clear description of decision problem including the objectives, the 1. Decision stakeholders, and alternatives, and whether the objective is to value or rank problem alternatives. Validation Validate the decision problem with decision makers and clinical experts Develop Report the sources used to identify criteria, the long list of criteria identified, 2. Selecting and the rationale for exluding criteria. criteria Report Report the final criteria list, including definitions and measurement scales. Validation Validate the criteria with stakeholders and against MCDA requirements (complete, non-redundant, non-overlapping, preferentially independent). 3. Measuring Methods Report and justify the sources uses to measure performance performance Reporting Report the performance matrix, showing performance of alternatives against criteria MCDA model Report the MCDA model (value measurement, outranking, reference), and 4. Scoring and justify in reference to the decision problem and stakeholder preferences. weighting Scoring / Justify the methods used for scoring and weighting with reference to the weighting objectives of the analysis and the stakeholders involved in the analysis. methods Reporting Report the values of scores and weights Validation Validate the meaning of scores and weights with stakeholders 5. Aggregation Reporting Report the aggregation function used in the analysis Validation Validate with stakeholders that the aggregate scores reflect how they expected their scores and weights to be used 6. Dealing with Report Report and justify variation (ranges, distribution) in parameter inputs, uncertainty including sources, and list all assumptions made (including structural assumption). Characterizing Consider the effects on the results of parameter uncertainty and uncertainty assumptions. Uncertainty At a minimum deterministic sensitivity analysis should be performed. Justify methods the method adopted to explore uncertainty with reference to the differences between techniques Characterizing Consider the effects on results of variation in scores and weights and heterogeneity performance measures between sub-groups. 7. Robustness The findings should be interpreted in light of the results of the validation Interpretation undertaken and the results of the analysis of the impacts of uncertainty and heterogeneity. Meaning The finding should be interpreted in light of the meaning of the particular scores and weights employed. 7 items with 21 sub-items: 1. Decision problem 2. Selecting and structuring criteria 3. Measuring performance 4. Scoring and weighting 5. Aggregation: analysis and report 6. Dealing with uncertainty 7. Interpretation Define and validate the decision problem, including the context, stakeholders and range of alternatives HTA, benefit-risk, clinical decisions, portfolio management Define and validate the preferred output of the MCDA experiment, i.e. ranking of alternatives or value functions weight / rank cost a c b a b c options Might be relevant for priority setting value Resource allocation decisions 8

9 Report the sources and justify methods for identifying, selecting and excluding criteria Summarize and structure the criteria in a value tree to address the decision problem. Avoid overlap and preferential dependence Determine clarity of criteria to stakeholders Validate the criteria and the value tree with stakeholders In hierarchies, criteria with many sub-criteria tend to be receiving more weight (Stillwell, 1987; Weber, 1988; Pöyhönen, 1997) Splitting-bias, i.e. if attribute presented in more detail it increases the weight it receives (Ha ma laïnen, 2003) decision decision Risk Outcome Cost Risk HrQoL 0.44 Pain Cost

10 Report the MCDA model and justify in reference to the decision problem and objectives of the study Determine and justify the population included for the elicitation of weights Facilitated group discussion vs. surveys Generalizability and reliability of the results Justify the method for weighting the criteria, i.e. pairwise, ranking, swing weights etc. Identify the range of performances of the alternatives on each criterion before scoring Validate the meaning of the weights and scores with stakeholders Construct a performance matrix or effects table providing the synthesized performance for each criterion Provide weights for each criterion being evaluated, including ranges Compute partial and overall scores, including their distributions Verify that the aggregate scores reflect the expectations of the stakeholders 10

11 Praveen Thokala University of Sheffield If a weighted-sum approach is used (as is often the case), criteria should meet following requirements: Completeness: The criteria should capture all factors important to measuring stakeholders objectives. Non-redundancy: Criteria should be removed if they are unnecessary Non-overlapping: Criteria should be defined to avoid double counting Preferential independence: The preference for a criteria (the weight it is given) should be independent of performance on another criteria. 11

12 Sources Mission statements Reviews of similar decisions Engagement with stakeholders: focus groups, surveys Stakeholders Decision makers Those whose preferences decision makers are concerned with Criteria should be defined clearly enough to be assessed. Expert consultation should be undertaken to help resolve any challenges defining and assessing criteria. Criteria should be defined in a manner than allows assessment in terms of absolute scales How many criteria should I include? As few criteria as is consistent with making a wellfounded decision. The MCDA should be sufficiently complex to answer the question. 12

13 Weighted sum approach V(a) = w i *v i (a) Use the scores and weights to estimate the overall value for each of the options Weights capture priorities or preferences between criteria. Scores capture priorities or preferences within a criterion. Many methods for eliciting scores and weights Need to ensure interpretation of weights and scores is a) understood by the participants and b) consistent with the objectives of the MCDA model. We classify the methods broadly into Direct methods Indirect (or choice based) methods 13

14 Scoring Direct rating of alternatives VAS AHP Category estimation Value function approach Bisection Difference standard sequence Developing the form of value function Weighting Direct rating VAS AHP SMART SMARTER SWING Combined weighting and scoring e.g. Points allocation Participants rank two or more real or hypothetical alternatives defined on some or all of the criteria. Weights and scores are derived from these rankings simultaneously using regression-based techniques Indirect methods include Conjoint Analysis (or DCE) PAPRIKA Revealed preference approaches have also been proposed to estimate decision makers preferences based on retrospective analysis of decisions 14

15 The most appropriate method will depend on the objective of the analysis, the time available to undertake the analysis, and the stakeholders Theoretical relevance: This is a function of whether the methods adopted comply with the assumptions of weighted sum approach i.e. scores need to have interval scale properties and weights need to represent the trade-offs Cognitive burden: Need to consider the burden placed on participants as it varies with methods Participants should be provided with sufficient information about the decision problem to ensure they are able to provide informed responses. Participants need to understand the meaning of scores and weights, especially how they comply with the needs of MCDA Participants need to be educated on how their inputs will be used. Also, they need to validate the MCDA results 15

16 Nancy Devlin Office of Health Economics 1. Should the criteria and weights used in MCDA by a given decision maker be the same for each decision, or chosen on a case-by-case basis? Best practice might depend on what type of health care decision MCDA is being applied to. e.g. benefit risk assessment: no budget constraint; relevant risks and benefits differ between each case. e.g. HTA; budget holder prioritisation frameworks: entail repeated decisions aiming to achieve efficiency from a limited budget. Consistency is key? 16

17 2. What should best practice guidelines say about whose weights are relevant? Essentially a normative question. Answer clear for some decision types: Shared Decision Making: weights should reflect the patient s preferences. Portfolio Optimisation: weights reflect the maximand of the decision maker (e.g profit) More complex for HTA, BRA, budget allocation: Patients? Committee members? The general public? What should we be recommending about this? Or should we remain agnostic? 3. Does best practice entail MCDA be used for all decisions or more selectively, e.g. only for complex decisions. If consistency of decisions is key, consistent application of MCDA suggested Selective use of MCDA reduces costs but how to select/recommend when it should be used? (eg. what is a complex decision?) 17

18 4. How should organisational resource constraints be reflected in our guidance on best practise in MCDA? Not helpful to identify best practice guidelines in terms of best technical approaches which, in practice, impose too great a burden (time, money) to implement. Best practice needs to be fit for purpose Should we also be identifying second best practice? eg. is partial use of MCDA, to generate a performance matrix for options, better than nothing at all? Thank you We welcome your questions and feedback 18