Overcoming psychological barriers to good decision-making in drug discovery

Transcription

1 Overcoming psychological barriers to good decision-making in drug discovery Keystone Symposium, Addressing the Challenges of Drug Discovery March 22 nd 2012 Matthew Segall - matt.segall@optibrium.com Optibrium, StarDrop, Auto-Modeller and Glowing Molecule are trademarks of Optibrium Ltd.

2 Overview Drug discovery productivity The challenge of decision-making in drug discovery The human factor cognitive biases Comparison with Evidence-Based Medicine (EBM) Common cognitive biases Confirmation bias Poor calibration Availability bias Excess focus on certainty Conclusion Guiding decisions to overcome biases 2

3 Drug Discovery Productivity Facts and figures abound for the decrease in productivity of pharmaceutical R&D Some debate is this because: We ve become less efficient at discovering NCEs, or All the low-hanging fruit have been discovered? In either case, improving productivity would be a good thing! 3

4 Challenges of Decision-Making in Drug Discovery Importance of multiple, sometimes conflicting, criteria to the success of a potential drug molecule Different degrees of importance of properties Lots of data Potentially large numbers of compounds Multiple properties Uncertain information Variability/error in data or predictions In silico predictions, in vitro assays and in vivo models are only approximations of the ultimate human target Long time scales Difficult to learn from mistakes (individual or organisation) 4

5 Cognitive Biases The human factor Psychological research shows that people are poor at making complex decisions Particularly involving risk/uncertainty System 1 vs. System 2 Gut instinct versus rational consideration Many examples, but we will focus on 4 common biases* Confirmation bias Poor calibration Availability bias Excess focus on certainty Contrast drug discovery with Evidence Based Medicine *Kahneman & Tversky, On the psychology of prediction. Psychological review, 1973, 80,

6 Confirmation Bias

7 Confirmation Bias People tend to look for evidence to support their hypotheses rather than refute them Psychological experiment by Peter Cathcart Wason in 1960* The sequence 2, 4, 6 obeys a rule what is it? To test your hypothesis, you can specify other sequences of three numbers and ask if they obey the unknown rule. When you re confident, you can announce what you think it is. The answer? Any ascending sequence! Self justification, overconfidence and premature closure * Wason, P.C. (1960) Quarterly Journal of Experimental Psychology, 12, pp

8 Confirmation Bias Implications In medicine: Study of diagnostic error* (90 injuries, including 33 deaths) Cognitive factors contributed to diagnostic error in 74% of cases: Premature closure, i.e. the failure to continue considering reasonable alternatives after an initial diagnosis was reached, was the single most common cause. In drug discovery: Projects failed too late Insufficiently wide search. *Graber, M.L. et al (2005) Diagnostic Error in Internal Medicine. Arch. Intern. Med. 165,

9 Confirmation Bias Example from a drug discovery project Project chemical space of >3100 compounds. First 200 progressed, shown in chronological order pki = 9.4 %F = 72 B:B = 0.01 pki = 8.9 %F = 3 B:B = 3.9 N=50 N=100 N=150 N=200 9

10 Confirmation Bias Possible Solutions EBM: Map of Medicine Visualisation of evidence-based pathway for common conditions See Drug discovery Balance quality with diversity when selecting compounds Libraries of evidence-based screening plans with interactive support for modification for different projects and therapeutic areas 10

11 Calibration Bias

12 Poor Calibration of Error/Risk People tend to be over-confident about their ability to estimate/predict Asked a group of experienced scientists to estimate the length of the Thames in kilometres. Answer could be any range in which they were 90% confident the correct answer lay. If perfect calibration, expect 90% of ranges would include correct range Only 20% of answers contained the correct value in the range! In medicine Poor balance of risks of inaction and action (e.g. use of biopsies) In drug discovery Underestimate risk late stage failures Inappropriate weight given to early screening results excess attrition and loss of opportunity 12

13 Poor Calibration Possible Solutions EBM For breast cancer radiographic screening in the UK there is a 'round robin' exchange of blinded test cases* Tracking and reporting critical outcome measures, such as sensitivity, specificity, size and stage of tumours detected, interval cancer rates, and time to recall and diagnosis, have been used in many countries to improve screening performance Drug discovery Training, e.g. anonymised cases for practice and feedback - to take the 'ego' out of decision making *Esserman, L. et al (2002) Improving the Accuracy of Mammography: Volume and Outcome Relationships. JNCI 94(5),

14 Availability Bias

15 Availability Bias People have a tendency to focus on the vivid or recent It has been estimated that following 9/11, over 1,500 additional people were killed in road accidents, due to increased road use as people avoided flying In medicine New clinicians have a tendency to consider rare and exotic diseases over more mundane explanations for symptoms the House effect In drug discovery Too much emphasis given to faint signs of issues, e.g. toxicity Excess attrition and loss of diversity opportunity cost 15

16 Availability bias How well does this assay conserve your options? You have purchased a series of compounds: Answer? You expect 1% of your compounds have a particular kind of toxicity You apply a screening method to all the compounds that is 90% reliable (both 90% sensitive and 90% specific) What percentage of the compounds that fail the screening genuinely have the toxicity? a) About 1% b) About 2% c) About 10% d) About 50% e) About 90% c) Of 1000 compounds, 990 x x 0.9 = 108 would be reported as toxic by the test, of which only 9 really are toxic. Neglect of the prior Prior What are appropriate priors? Calibration bias: not necessarily good at estimating priors 16

17 Example Application Screening Strategy* Two screens for toxicity: in silico and in vitro In silico: cost 1, accuracy 80% In vitro: cost 100, accuracy 95% Cost to prove safety 5,000 Net value of safe compound 10,000 5 Possible screening strategies *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 17

18 Example Application Screening Strategy Two screens for toxicity: in silico and in vitro 5 Possible screening strategies: Double Filter In silico test Pass Fail In vitro test Pass Fail Outcome Safe P=99.4% v=9899 Unsafe P=0.6% v=-5101 Reject P=13.7% v=-101 Reject P=38.0% v=-1 *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 18

19 Example Application Screening Strategy Two screens for toxicity: in silico and in vitro 5 Possible screening strategies: Sentinel Outcome Safe P=90.3% v=9999 Unsafe P=9.7% v=-5001 In silico test Pass Fail In vitro test Pass Fail Outcome Safe P=91.7% v=9899 Unsafe P=8.3% v=-5101 Reject P=61.8% v=-101 *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 19

20 Example Application Screening Strategy Two screens for toxicity: in silico and in vitro 5 Possible screening strategies: In Silico Only In silico test Pass Fail Outcome Safe P=90.3% v=9999 Unsafe P=9.7% v=-5001 Reject P=38.0% v=-1 *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 20

21 Example Application Screening Strategy Two screens for toxicity: in silico and in vitro 5 Possible screening strategies: In Vitro Only In vitro test Pass Fail Outcome Safe P=97.8% v=9900 Unsafe P=2.2% v=-5100 Reject P=32.0% v=-100 *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 21

22 Example Application Screening Strategy Two screens for toxicity: in silico and in vitro 5 Possible screening strategies: No Screen Outcome Safe P=70% v=10000 Unsafe P=30% v=-5000 *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 22

23 Example Application Screening Strategy Parameters: In silico: cost 1, accuracy 80% In vitro: cost 100, accuracy 95% Cost to confirm safety 5,000; Net value of safe compound 10,000 Strategy Value (Prior for risk 30%) Value (Prior for risk 40%) Double filter Sentinel In silico only In vitro only No screen *Segall and Chadwick, Making Priors a Priority, J. Comp.-Aided Mol. Des., 24, , (2010) 23

24 Possible Solutions Quantitative Analysis of Screening Options Strategy value In vitro cost/ compound value Downstream cost/ net value For an interactive example visit 24

25 Excess Focus on Certainty

26 Excess Focus on Certainty People tend to seek more and more certainty even when it adds little value at high cost Headlines such as XXX increases the risk by 50%! What was the initial risk? Human decision-makers are inconsistent in applying the rules they describe if questioned on the basis for their decisions*: the overwhelming conclusion, including studies of clinical judgment, was that the linear model of the judge s behaviour outperformed the judge. *Goodwin and Wright Decision Analysis for Management Judgment (3 rd ed.), Wiley p (2004) 26

27 Excess Focus on Certainty Implications In medicine Clinical guidelines difficult to agree and use Problems reassuring patients. In drug discovery Inefficient use of resources when screening across multiple risk factors 27

28 Possible Solution Multi-parameter optimisation* E.g. Probabilistic scoring *Segall, Multi-Parameter Optimization..., Curr. Pharm. Des., 18, (2012) 28

29 Possible Solution Multi-parameter optimisation Score Property data Experimental or predicted Criteria for success Relative importance Uncertainties in data Experimental or statistical Score (Likelihood of Success) Confidence in score Error Bottom bars show 50% may be Data confidence do not rejected in with separate overall score these, as confidence error bars overlap Best Worst 29

30 Illustrative Example* In vitro data being generated Potency Selectivity Solubility Microsomal stability (rat and human: RLM and HLM) Original process focused on potency and selectivity, filtering compounds that did not meet requirements Results Low but prolonged activity after IP dosing No correlation between in vitro and in vivo potency Problems with solubility and metabolic stability *Segall et al., Focus on success... Expert Opin. Drug. Metab. Toxicol., (2006) 30

31 Comparison of Strategies Potency and selectivity No uncertainty - filter Potency and selectivity Consider uncertainty All properties Consider uncertainty 31

32 Conclusions Drug discovery scientists are human! Cognitive biases can be a barrier to good decisions and hence impact drug discovery productivity Can be addressed through a combination of training and tools to apply decision-analysis approaches to guide decisions A critical issue priors What are appropriate priors for common drug discovery risks? Need to pool information on priors and method reliability Chadwick and Segall, Overcoming psychological barriers to good discovery decisions. Drug Discov. Today., 15, (2010) 32

33 Acknowledgements Tessella Andrew Chadwick - Andrew.Chadwick@tessella.com Rob Rowley Optibrium Ed Champness Chris Leeding Iskander Yusof James Chisholm Interesting light reading Ben Goldacre (2008) Bad Science, Fourth Estate (paperback) Dan Gardner (2009) Risk the science and politics of fear, Virgin Publishing (paperback) 33

34 Anchoring* Question: Estimate the percentage of African nations that are members of the United Nations. First spin a wheel to generate random number between 1 and 100 and ask if estimate was higher or lower If random number was 10: Median estimate 25% If random number was 65: Median estimate was 45% *Tversky &Kahneman, Judgment under Uncertainty: Heuristics and Biases. Science, 185, (1974) 34