EXPECTED VALUE OF PERFECT INFORMATION: ACTIVE LEARNING THROUGH USER-FRIENDLY COMPUTATIONS, DISPLAYS, AND ASIA PACIFIC APPLICATION DISCUSSIONS Jonathan D. Campbell, PhD 1 R Brett McQueen, PhD 1 Jonathan H. Watanabe, PharmD, PhD 2 Nathorn Chaiyakunapruk, PharmD, PhD 3, 1. University of Colorado, Aurora, CO 2. University of California, San Diego, CA 3. Monash University, Kuala Lumpur, Malaysia Workshop Objectives Evidence for cost-effectiveness and value of information Review theory and concepts of value of information Expected Value of Perfect Information (EVPI) active learning Gain understanding through a workshop exercise Display EVPI case Discuss EVPI applications 1
What is the question? Is the study population consistent with your own? Are the outcomes for effect size measurement useful? Are you interested in an idealized efficacy analysis to discern the possibility of benefit? Are you interested in an effectiveness analysis that integrates a real-world population? What evidence is missing in the available literature? Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy Turner et al. N Engl J Med 2008;358:252-60. What is good evidence? Although studies can arrive at the same conclusion, the methods of each study will differentiate the validity Large Cohort Study of Drug A versus B Users follows patients to determine risk of rare side effect (<1 %) Large Case-Control Study of those with and without the rare side effect (<1 %) to determine probability of exposure Relative Risk of 1.8 Odds Ratio of 1.8 Relative Risk of 1.8 The cohort study would generally be afforded more validity as it compared an exposed and NON-exposed cohort to measure risk excess. The case-control study does not track exposed and non-exposed to event, but only probability of exposure. 2
Pyramid of Evidence Sackett DL, Straus SE, Richardson WS, et al. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh: Churchill Livingstone, 2000.) Validated and Systematic Scoring of Clinical Trials Studies: Jadad Scores 3 items Maximum of 5 points > 10, 000 Citations Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996 Feb;17(1):1-12. PubMed PMID: 8721797. 3
Validated and Systematic Scoring of Observational Studies: Newcastle-Ottawa Scale Uses a star system on 3 categories: 1. Selection 2. Comparability 3. Outcome or Exposure Outcome for Cohort Studies Exposure for Case-Control Studies Wells G (2011) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Available: http://www.ohrica/programs/clinical_epid emiology/oxford.asp. Selection 1. Representativeness of the exposed cohort a) truly representative of the average (describe) in the community b) somewhat representative of the average in the community c) selected group of users eg nurses, volunteers d) no description of the derivation of the cohort 2. Selection of the non exposed cohort a) drawn from the same community as the exposed cohort b) drawn from a different source c) no description of the derivation of the non exposed cohort 3. Ascertainment of exposure to implants a) secure record (eg surgical records) b) structured interview c) written self report d) no description 4. Demonstration that outcome of interest was not present at start of study a) yes b) no 4
Comparability 1. Comparability of cohorts on the basis of the design or analysis a) study controls for (select the most important factor) b) study controls for any additional factor (This criteria could be modified to indicate specific control for a second important factor.) 1. Assessment of outcome a) independent blind assessment b) record linkage c) self report d) no description Outcome 2. Was follow up long enough for outcomes to occur a) yes (select an adequate follow up period for outcome of interest) b) no 3. Adequacy of follow up of cohorts a) complete follow up - all subjects accounted for b) subjects lost to follow up unlikely to introduce bias - small number lost - > % (select an adequate %) follow up, or description of those lost) c) follow up rate < % (select an adequate %) and no description of those lost d) no statement 5
Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Ever Use) Cohort Studies Selection Comparability Outcome Lauritzen / 83 Wilson / 85 Petitti / 87 Henderson / 91 Lafferty / 94 Folsom / 95 Ettinger / 96 Wolf / 96 0.01 0.1 1 10 Was there some form of sensitivity analysis to provide an uncertainty band? Did they provide a quantitative result of the robustness of their study findings? Did they test other regression models? Did they account for missing data? 6
Words of Wisdom As a medical statistician, I am appalled by the large number of irreproducible results in the medical literature. There is a general, and likely correct, perception that this problem is associated with statistical, as opposed to laboratory, research. I am convinced however that results of clinical and epidemiological investigations could become more reproducible if only the investigators would apply more rigorous statistical thinking and adhere more closely to well established principles of the scientific method --Norman Breslow All models are wrong, but some are useful --George Box Theory and Concepts of Value of Information 7
Welfare Economics & Bayesian Decision Theory Decision Analytic Model Mean Net benefit Uncertainty in Net Benefit Q1: Adopt Intervention with Present Information? Expected Value of Perfect Information Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics 18 (1999): 341-364. Q2: Generate Future Evidence? Decision Analytic Model (e.g., Population, intervention, comparator(s), outcome(s), time that best represent disease state) Maximize Social Welfare Function (normative weights included) Mean Net benefit Mean Net Benefit = QALYs*Willingness to pay - Costs Positive analysis Q1: Adopt Intervention with Present Information? Answer = maximum mean net benefit 8
Welfare Economics & Bayesian Decision Theory Decision Analytic Model Mean Net benefit Uncertainty in Net Benefit Q1: Adopt Intervention with Present Information? Expected Value of Perfect Information Q2: Generate Future Evidence? Decision Analytic Model Uncertainty analysis (i.e., probabilistic sensitivity analysis) Uncertainty in Net Benefit Decision Uncertainty: probability of error Perfect information (infinite sample) Present information = E(max net benefits for each iteration) max of E(net benefits of all iterations) Expected Value of Perfect Information Upper bound on value of generating future evidence Q2: Generate Future Evidence? 9
EVPI Active Learning EVPI Active Learning Definition: E(max net benefits for each iteration) max of E(net benefits of all iterations) Inputs to calculate EVPI: Willingness to pay (how much society is willing to pay for an extra unit of health i.e., QALY) QALYs for each intervention and Monte Carlo iteration Costs for each intervention and Monte Carlo iteration Net benefit = (QALYs)*willingness to pay - Costs 10
$25,000 $50,000 $75,000 $100,000 $125,000 $150,000 $175,000 $200,000 $225,000 $250,000 $275,000 $300,000 $325,000 $350,000 $375,000 $400,000 $425,000 $450,000 $475,000 $500,000 EVPI Solution Net Benefit: A Net Benefit: B Max Net Benefit Choose A or B? $ 25,000.00 $ - $ 25,000.00 A $ 75,000.00 $ 50,000.00 $ 75,000.00 A $ (50,000.00) $ 25,000.00 $ 25,000.00 B $ 50,000.00 $ 50,000.00 $ 50,000.00 tie $ 25,000.00 $ 125,000.00 $ 125,000.00 B Expected Net Benefit: A Expected Net Benefit: B Expected Max Net Benefit $ 25,000.00 $ 50,000.00 $ 60,000.00 EVPI $ 10,000.00 Display EVPI Case $40,000 $35,000 $30,000 $25,000 $20,000 $15,000 $10,000 $5,000 $- EVPI across varying WTP values Willingness to pay (US dollars) 11
Reason for a Data Table The EVPI example was based on one willingness-to-pay value What if we wanted to know what EVPI was at $50,000/QALY or $75,000/QALY? To answer this we could manually calculate EVPI for different values of willingness-to-pay Excel can automatically calculate this and record the values in one table using a data table Value of Information Applications 12
Examples of Studies using EVPI in Asian counties Accepted ceiling value = 270,000 THB/QALY Future research is not required!!! 13
A future research on vaccine efficacy might be a priority. 14
VOI Studies in Asian Countries Only a few studies have performed VOI in Asian countries Even though the VOI findings provided clear messages, it was not clear whether the findings have been used for guiding future research direction. There has been no mention of VOI as part of pharmacoeconomics and HTA guidelines in AP region Challenges will be on how to use VOI analysis for research prioritization 15
Current State of VOI in Health Care Research Prioritization (PCORI, 2012) Very small number of research informed by VOI studies 16
Challenges to use VOI studies for research prioritization (PCORI,2012) Resources Personnel Time Computing resource Scope of VOI Prioritizing across disease areas (Challenging) Prioritizing within a condition (Possible) Prioritizing specific CER (Most common application of VOI) Prioritizing attributes of patient experience common to multiple conditions Challenges to use VOI studies for research prioritization (PCORI,2012) Stakeholder Engagement Lack of familiarity with the methodology Time of VOI Heterogeneity and Equity Effect of Resolving Uncertainty on Patient and Provider Choices 17
EXPECTED VALUE OF PERFECT INFORMATION: ACTIVE LEARNING THROUGH USER-FRIENDLY COMPUTATIONS, DISPLAYS, AND ASIA PACIFIC APPLICATION DISCUSSIONS Jonathan D. Campbell, PhD 1 R Brett McQueen, PhD 1 Jonathan H. Watanabe, PharmD, PhD 2 Nathorn Chaiyakunapruk, PharmD, PhD 3, 1. University of Colorado, Aurora, CO 2. University of California, San Diego, CA 3. Monash University, Kuala Lumpur, Malaysia Jon.Campbell@ucdenver.edu; Robert.McQueen@ucdenver.edu; Jhwatanabe@ucsd.edu; nui_nathorn@yahoo.com 18