How to incorporate economic evaluations in clinical practice guidelines: a practical workshop on research methods

Transcription

1 How to incorporate economic evaluations in clinical practice guidelines: a practical workshop on research methods Please boot your laptop/tablet and go to: PubMed.gov Shortlink: bit.ly/pubmed ISPOR November 2th 2016 How to incorporate economic evaluations in clinical practice guidelines: a practical workshop on research methods Ghislaine van Mastrigt Ben Wijnen Frederick Thielen Jos Kleijnen Silvia Evers ISPOR November 2th

2 Background check!? HTA experience? Systematic review? Clinical practice guidelines? Funding 2

3 Program of workshop Session lead: Silvia Evers Time Topic Step Presenter Background Overall approach Initiating Systematic Review of economic evaluations Ghislaine van Mastrigt Identifying full economic evaluations 2 Frederick Thielen Data extraction and Quality appraisal 3 Ben Wijnen Reporting and discussion & recommendations Advice 4-5 Ghislaine van Mastrigt Jos Kleijnen Questions from the audience All Background Overall approach (steps 1-5) Initiating a Systematic Review of economic evaluations (step1) Ghislaine van Mastrigt ISPOR November 2th

4 Background What are Clinical Practice Guidelines? Clinical Practice Guidelines are statements that include recommendations intended to optimize patient care that are informed by systematic review of evidence and an assessment of benefits and harms of alternative care options (Institute of Medicine 2011) Background Taking into account economic evidence in evidence based medicine new / clinical practice guidelines? Archie Cochrane ( ) No! Alan Williams ( ) If we are ever going to get the optimum results from our national expenditures on NHS we must finally be able to express the results in the form of benefits and the costs to the population of a particular types of activity, and increased benefit that would be obtained if more money would be made available. (Cochrane 1972) Current EBM is not enough Costs should be considered systematically alongside effectiveness Cost and Effect together in decision analytic framework All cost-effective treatments should be free in NHS, etc (Williams 1997) 4

5 Background Why incorperate EEs into Clinical Practice Guidelines? Increase in health care expenditures all over the world Newly development interventions are often more expensive as the existing ones Health care budget is limited Tension between financial and clinical possibilities Clinical practice guidelines recommend about most effective treatments for a specific disease, why not also take into account the best value for money (cost-effective treatments)? Background Why incorperate EEs into Clinical Practice Guidelines? Guidance for Clinical Practice Guidelines AGREE II: The potential resource implications of applying the recommendations have been considered. Guideline International Network: The recommendations should be supported by careful consideration of evidence;.costs when possible; resource and feasibility issues; NICE: Guideline recommendations should be based on the estimated costs of the interventions or services in relation to their expected health benefits (that is, their 'cost effectiveness'). 5

6 Overall 5-step approach Step 1: Initiating a Systematic Review of economic evaluations Step 2: Identifying full economic evaluations Step 3: Data extraction and quality appraisal Step 4: Reporting of results Step 5: Discussion and interpretation of results General recommendations Basic knowledge of HTA Drummond et al Country specific HTA guidelines (see ISPOR website) Basic knowledge CPG Use AGREE II Tool: CPG development Check country specific regulations Minimize biases Steps 2 and 3: by two reviewers independently Statements Use PRISMA statement: reporting SRs Use CHEERS statement: reporting EEs 6

7 Step 1 Initiating a SR of EEs 1. Compose multi-disciplinairy projectteam Clinician(s) HTA & SR & quantitative methods CPG development Information specialist Patient & Public views 2. Identify relevant topics Consult different stakeholders Perform scoping review 1) SR of clinical effectiveness 2) SR of cost effectiveness Step 1 Initiating a SR of EEs 3. Write protocol SR of EE PRISMA-P statement ( 4. Publish protocol SR of EE ( ( 7

8 Identifying full economic evaluations (step 2) Frederick Thielen ISPOR November 2th 2016 Step 2 Identifying full EEs Selecting databases and developing a search strategy A very short introduction and practical example End 2014 Health Economic Evaluation Database (HEED) ceased publication March 2015 NHS Economic Evaluation Database (NHS EED) is no longed updated Today: merely an archive 8

9 Step 2 Identifying full EEs Original research Gray literature Bibliographic databases Basic Specific Websites HTA reports CRD database TODAY Optional Employ search strategy Employ search terms Step 2 Identifying full EEs Basic Specific Optional Databases Medline Embase CINAHL (nursing) CADTH HTA Database As archive only! NHS EED PEDro (physiotherapy) ISPOR website Econlit (EBSCO) PsycINFO (psychology) Cochrane Colloquium Web-of- Science 9

10 Step 2 Identifying full EEs Database: Medline Interface: Databases Cave! Database User interface Step 2 Identifying full EEs Workshop element 1 Databases Example Ketogenic diet in Epilepsy Please go to PubMed.gov Shortlink: bit.ly/pubmed 10

11 Step 2 Identifying full EEs Search terms Development: hours Health / disease concept Intervention concept Economics concept Already existing reviews Existing search strategies If above not applicable: find your own Search filters Search strategy Step 2 Identifying full EEs Workshop element 2 Task: Find all economic evaluations that examine the costeffectiveness of ketogenic diet in patients with epilepsy. Source: 11

12 Step 2 Identifying full EEs Task: Find all economic evaluations that examin the costeffectiveness of ketogenic diet in patients with epilepsy. Patients Epilepsy AND The PICO scheme Intervention Comparator Ketogenic diet AND not applicable AND Outcome Economic Evaluation Step 2 Identifying full EEs Identifying search terms: advanced Two major thesauri: 1. MeSH (Medline) 2. Emtree (Embase) Three interfaces: 1. PubMed (MeSH) 2. Ovid (MeSH and Emtree) 3. Embase (Emtree) 12

13 Step 2 Identifying full EEs Identifying search terms: advanced Step 2 Identifying full EEs Search filters Advice: Sensitive rather than precise filter Can be tested according to: sensitivity, precision, and specificity High level of sensitivity lower level of precision (and vice versa) Find search filters on the ISSG* website: * InterTASC Information Specialists' Sub-Group 13

14 Step 2 Identifying full EEs Search filters Search filter for economics in MEDLINE with best balance of sensitivity and specificity, taken from Wilczynski et al. [1] cost*[title/abstract] OR "costs and cost analysis"[mesh:noexp] OR cost benefit analys*[title/abstract] OR cost-benefit analysis[mesh Term] OR health care costs[mesh:noexp] [1] Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-Oatley AE, HSR Hedges team. Optimal search strategies for detecting health services research studies in MEDLINE. Canadian Medical Association Journal 2004;171(10): Step 2 Identifying full EEs Patients Epilepsy OR epilepsy[mesh] epilepsy[tiab] epileps*[tiab] epilept*[tiab] seizures[mesh] seizures[tiab] seizure[tiab] (convulsion OR convulsions[tiab]) AND The PICO scheme Intervention Comparator Ketogenic AND not applicable diet Synonyms / alternative keywords OR "ketogenic diet"[mesh] ( "ketogenic"[tiab] AND "diet"[tiab]) "ketogenic diet"[tiab] (ketogen*[tiab] AND diet[tiab]) "diet therapy"[mesh] "diet therapy"[tiab] AND Outcome Economic Evaluation Search filter taken from Wilczynski et al. [1] cost*[tiab] [1] Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-Oatley AE, HSR Hedges team. Optimal search strategies for detecting health services research studies in MEDLINE. Canadian Medical Association Journal 2004;171(10): "costs and cost analysis [MeSH:noexp] cost benefit analys*[tiab] cost-benefit analysis[mesh] health care costs [MeSH:noexp] 14

15 Step 2 Identifying full EEs Step 3: Data extraction sheet (3.1) and risk of bias assessment and transferability (3.2) Ben Wijnen ISPOR November 2th

16 Step 3.1 Data extraction For a SR-EE a tailored data extraction sheet needs to be developed. Inclusion of items depends on the research question or study objective (defined in protocol) Several example data extraction forms available from the literature, which can be classified in two groups General study characteristics Study methods and outcomes Step 3.1 Data extraction General study characteristics First author and year of publication Sources of funding Competing interests Publication type Setting Patient characteristics Type of intervention Control treatment Eligibility criteria Study perspective Type of EEs Analytic approach Methods and outcomes of economic evaluations Time frame of the analysis (time horizon) Discount rates Inflation rate Reference year Type and category of costs Data source of resource use Methods for identifying resource use Assumptions of the measurement of resources Costs (currency or in converted currency) Methods of used to calculate unit costs Data source of effects Methods of measurement of effects Methods of valuation of effects Incremental cost-effectiveness ratios Analyses of uncertainty (e.g. sensitivity analyses) Outcome(s) of analyses of sensitivity analyses Authors conclusions 16

17 Step 3.1 Data extraction Data extraction sheet Step 3.1 Data extraction General and RCT-related and economic characteristics Author Year Disease N Intervention Trial- or modelbased EE de Kinderen et al. Etc. Etc Refractory epilepsy 48 Ketogenic diet Intervention costeffective No, based on cost per QALY Trialbased EE Comparator(s) Outcome measure(s) Perspective Care as usual QALY gain healthcare perspective Study methods and outcomes Author Year Disease Main outcome ICER/ICUR de Kinderen et al. Etc. Etc Refractory epilepsy The results show that the KD reduces seizure frequency. The study did not find any improvements in quality of life and, therefore, unfavorable cost per QALY ratio s resulted. 1) CEA: per responder; 2) CUA: Not presented but 5% probability of being cost-effective at the ceiling ratio of 50,000 per QALY 17

18 Step 3.1 Data extraction Tips: Include risk of bias and transferability checklists in the data extraction sheet. It is recommended to use a picklist to choose the different answers. Describe items to increase transparency of scoring. Highly recommended to pilot the extraction sheet on user-friendliness and completeness (i.e. few sample studies) Recommendation to facilitate interpretation of the results: Disaggregated presentation of the results Incremental cost-effectiveness ratios Step 3.2 Risk of bias assessment 18

19 Step 3.2 Risk of bias assessment Economic evaluations are prone to bias (pre-trial and during-trial biases) For example: biases as a result of narrow perspective inefficient comparator cost measurement omission or inappropriate discounting A lot of risk of bias assessment or quality checklist for economic evaluations have been developed the past decades (11 checklist included) reporting quality; For reporting CHEERS checklist (Husereau et al. 2013) Be aware: most risk of bias checklists are designed for full-economic evaluations Step 3.2 Risk of bias assessment Risk of Bias assessment Use CHEC-extended or BMJ checklist (Evers et al. 2005; Drummond et al. 1998). For model-based EE and <10 studies: Phillips et al checklist. For model-based EEs and >10 studies, the ISPOR checklist (Caro et al. 2014). Specific for CPG Full EEs should be preferred over partial EEs at all times. However, partial EEs may represent important intermediate stages in our understanding and therefore might be convenient. 19

20 Step 3.2 Risk of bias assessment Risk of Bias assessment Use CHEC-extended or BMJ checklist (Evers et al. 2005; Drummond et al. 1998). Focus of today For model-based EE and <10 studies: Phillips et al checklist. For model-based EEs and >10 studies, the ISPOR checklist (Caro et al. 2014). Specific for CPG Full EEs should be preferred over partial EEs at all times. Partial EEs may represent important intermediate stages in our understanding and therefore might be convenient. Step 3.2 Risk of bias assessment CHEC-extended items (Evers et al. 2005; Odnoletkova et al. 2014) 1 Is the study population clearly described? 2 Are competing alternatives clearly described? 3 Is a well-defined research question posed in answerable form? 4 Is the economic study design appropriate to the stated objective? Are the structural assumptions and the validation methods of the model 5 properly reported? Is the chosen time horizon appropriate in order to include relevant 6 costs and consequences? 7 Is the actual perspective chosen appropriate? 8 Are all important and relevant costs for each alternative identified? 9 Are all costs measured appropriately in physical units? 10 Are costs valued appropriately? 11 Are all important and relevant outcomes for each alternative identified? 12 Are all outcomes measured appropriately? 13 Are outcomes valued appropriately? Is an appropriate incremental analysis of costs and outcomes of 14 alternatives performed? 15 Are all future costs and outcomes discounted appropriately? Are all important variables, whose values are uncertain, 16 appropriately subjected to sensitivity analysis? 17 Do the conclusions follow from the data reported? Does the study discuss the generalizability of the results to other 18 settings and patient/client groups? Does the article/report indicate that there is no potential conflict of 19 interest of study researcher(s) and funder(s)? 20 Are ethical and distributional issues discussed appropriately? General framing Methodological assumptions Uncertainty Conclusions Ethical concerns 20

21 Step 3.2 Risk of bias assessment Research question Population Intervention & control EASY? Step 3.2 Risk of bias assessment Q: Is the chosen time horizon appropriate in order to include relevant costs and consequences?. A: No, a four-months time horizon is too short; for a societal perspective a time frame of at least one year is generally accepted. Assessment instructions available, use them 21

22 Step 3.3 Transferability Step 3.3 Transferability Transferability Transferability: the extent to which the results of a study hold true for a different population or setting Generalizability: the extent to which the results of a study can be generalized to the population from which the sample was drawn Often used interchangeably 22

23 Step 3.3 Transferability Transferability Transferability: the extent to which the results of a study hold true for a different population or setting Generalizability: the extent to which the results of a study can be generalized to the population from which the sample was drawn Often used interchangeably Step 3.3 Transferability Transferability It is important to discuss the transferability of a particular study with clinicians as clinical practice might vary between countries Check differences in pharmacoeconomic guidelines between countries ( 23

24 Step 3.3 Transferability General knock-out criteria: Example: Welte et al. checklist to determine the transferability of EE of the ketogenic diet from the Netherlands to the UK setting (based on de Kinderen et al. 2016) General knock-out criteria 1. Evaluated technology comparable? Correspondence between study (the Netherlands) and decision country (UK) NA Passed 2. The comparator is comparable? NA Passed 3. Acceptable quality? NA Passed Step 3.3 Transferability Specific criteria: Correspondence between study (the Netherlands) and decision country (UK) ICER of decision country based on ICER of study country is: Methodological characteristics Perspective Low(healthcare vs societal) Too low or too high Discount rate Medium (3.5% UK vs 4% NL) Unbiased (short time horizon) Medical cost approach High Unbiased Productivity cost approach Low (friction costs method in NL Too low or too high vs no productivity costs in UK Healthcare system characteristics Absolute and relative prices in health High Unbiased care Practice variation Medium Too low or too high Technology availability Recalculate using health care High perspective Unbiased Population characteristics Disease incidence/prevalence High Unbiased Case-mix High Unbiased Life expectancy High Unbiased Health-status preferences High Unbiased Acceptance, compliance, incentives to High Unbiased patients Productivity and work-loss time High Unbiased Disease spread High Unbiased 24

25 Step 3 General remarks Tips: - Raters are a relevant source of variability. - It is important to have multiple raters (at least two) so that discrepancies can be resolved through consensus meetings. - A few studies (i.e. two or three) should be used to pilot the assessment between raters discuss discrepancies uniform assessment strategy. Reporting (step 4) Discussion and interpretation of results (step 5) Ghislaine van Mastrigt ISPOR November 2th

26 Step 4 Reporting of results Result section and data synthesis Convert different currencies reported to a common currency Use the same year as reference Graphic presentation of the data if common metrics (e.g. QALYs, costs) Trial-based EEs: GRADE evidence profiles and Summary of Findings tables Model-based EEs: Summary of Findings tables Meta-analysis is not recommended Step 5 Discussion and interpretation of results Discussion section Summary of evidence Heterogeneity Previous SR-EE findings in relation to current SR-EE findings Study limitations & strengths Time frame for update of SR General conclusions Recommendations for further research Source of funding Conflict of interest 26

27 Step 5 Discussion and interpretation of results Recommendations Check the following factors 1. Based on the risk of bias assessment, what is quality of the EE? 2. Is the new intervention cost-effective? 3. Are the outcomes uncertain and/or variable? 4. Is balance between health benefits, side effects and risk discussed? 5. Discuss transferability of study findings Intervention in CPG country Comparator in CPG country Quality of the study 6. Discuss generalizability of study findings 7. Discuss implementation problems Finally Discuss the overall strength of evidence Conclusion Five-step approach to incorporate EEs in CPGs Facilitate the incorporation of EEs in CPGs Increase transparancy and standarization of performing SRs of EEs Decrease workload of guideline developers Step 1: Initiating a SR of EEs Step 2: Identifying full EEs Step 3: Data extraction Step 4: Reporting of results Step 5: Discussion and interpretation of results 27

28 Discussion Jos Kleijnen ISPOR November 2th 2016 Systematic reviews of economic data We do not want an average of all studies found. Instead we look for the: Most representative information Most recent information Most valid information and Most precise information However, we do adhere to the principles of systematic reviews for searching, quality assessment, procedures, etc. 28

29 Questions? The series has been accepted for publication in Expert Review of Pharmacoeconomics & Outcomes Research How to Prepare a Systematic Review of Economic Evaluations for Informing Evidence-based Healthcare Decisions: A five-step approach (Part 1 of 3), van Mastrigt GAPG, Mikael Hiligsmann, Pieter Broos/Toon Lamberts, Silvia Evers, Marian Majoie. How to Prepare a Systematic Review of Economic Evaluations for Informing Evidence-based Healthcare Decisions: Database selection and search strategy development (Part 2 of 3) Thielen, W.F.; van Mastrigt, GAPG; Burgers, LT; Bramer, WM ; Majoie, HJM; Evers, SMAA; Kleijnen, J. How to Prepare a Systematic Review of Economic Evaluations for Informing Evidence-based Healthcare Decisions: Data extraction, risk of bias, and transferability (Part 3 of 3) Wijnen, BFM; van Mastrigt, GAPG; Redekop WK; Majoie, HJM; de Kinderen, RJA; Evers, SMAA. Thank you for your attention! 29