The use of integrated in silico solutions under the proposed ICH M7 guidelines

Transcription

1 The use of integrated in silico solutions under the proposed ICH M7 guidelines SOT Phoenix, March 2014 Dr Chris Barber Director of Science

2 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment from 2 predictions Avoiding the problem GTIPurge project

3 What does M7 cover? identification categorisation qualification Control of mutagenic impurities to limit potential carcinogenic risk Harmonises guidelines FDA, EMA, Japan Recognises the primacy of the Ames assay Applies to new drug products existing ones if change to route formulation dosing regime indication biologics peptides anticancer treatments existing excipients

4 M7 guides the decision process identification categorisation qualification known mutagen known carcinogen unknown carcinogen Class 1 Control < compound-specific limit Class 2 Control < generic or adjusted TTC limits contains structural alert Alert not in API No mutagenicity data Same as API which tested non-mutagenic Class 3 Control < generic or adjusted TTC limits or run Ames Class 4 Treat as non-mutagenic no structural alert or negative Ames Class 5 Treat as non-mutagenic

5 Focussing on the identification step Evaluate drug substance, impurities, degradants, (metabolites), intermediates Databases, in-house, literature.. 2 x in silico QSAR Leadscope Multicase Known mutagen Predicted positive Predicted negative Known non-mutagen Expert Review Ames test Expert Review Limit according to TTC or present purge argument for absence GTI Purge Tool Treat as nonmutagenic

6 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment from 2 predictions Avoiding the problem GTIPurge project

7 Vitic Nexus an authoritative toxicity database Vitic Nexus is a repository of toxicological data Data donated by members Curated and augmented by expert scientists Genotoxicity records In vitro data In vivo data Overall call 146,444 records, 9,014 compounds 10,157 records, 2,658 compounds 15,289 records, 8,510 compounds Contains public datasets and literature including o Benchmark, CGX, ISSSTY, IUCLID o FDA CDER & CFSAN, o JETOC (Japanese Chemical Industry Ecology-Toxicology..) o IARC, JETOC, NIHS, NTP, SCCP, SIDS Members also store their own data in Vitic Nexus

8 Data sharing consortia Lhasa facilitate pre-competitive data sharing Members of these consortia also see Aromatic amines 1,664 records 145 compounds Intermediates (includes boronic acid sub-group) 13,834 records 910 compounds Excipients 2,286 records 764 compounds

9 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment from 2 predictions Avoiding the problem GTIPurge project

10 in silico predictions for M7 Use models that predict Ames outcomes 2 complementary methods should be applied One expert rule-based One statistical-based Models should follow OECD Principles for QSAR The absence of alerts from both is sufficient to conclude that the impurity is of no concern Expert review is needed to provide additional evidence for any prediction and to explain conflicting results

11 Expert rule based systems Are not statistical systems Rules created by scientists with expertise in chemistry and the endpoint the collective knowledge of a group of experts Comprises of a knowledge base and a reasoning (inference) engine allows extrapolation based upon expert knowledge can deal with contradictory facts and uncertainty explains why and how a conclusion is reached

12 Expert rule based systems Should provide sufficient information to support expert analysis mechanistic, supporting data, expert opinion, data interpretation, measure of accuracy, validation stats, relevant examples.. Can be trained on confidential data without disclosing those underlying compounds This is really important for a robust model Can Public Data Improve Mutagenicity Predictions for Proprietary Compounds? o March 27 th am - Poster No / 444

13 Derek Nexus an expert knowledge base Derek Nexus is built using public & confidential data in collaboration with regulators and our members Comprises of >110 alerts for mutagenicity Can be further customised by members with private knowledge Also predicts many other key endpoints including skin toxicity, hepatotoxicity, cardiotoxicity, kidney toxicity, phototoxicity, phospholipidosis, reprotoxicity Addressing toxicity risk when designing and selecting compounds in early drug discovery o In press, Drug Discovery Today, Jan 2014

14 Structure 2. Unambiguous algorithm 3. Applicability domain 5. Mechanism 4. Performance 1. Defined endpoint References Prediction Likelihood Key examples

15 Accuracy of Derek predictions Derek Nexus provides a level of confidence (likelihood) for each prediction This correlates well with accuracy CFSAN Hansen NTP % Active compounds Assessing confidence in predictions made by knowledge-based systems. Judson et. al. Toxicology Research, 2013, 2, 70

16 Derek Nexus an expert knowledge base Derek is the preferred system for our members In silico methods combined with expert knowledge rule out mutagenic potential of pharmaceutical impurities: An industry survey Regulatory Toxicology and Pharmacology, 2012, 62, Pfizer, Novartis, GSK, AZ, Lilly, Hoffmann-La Roche, Covance, Merck, J&J Use of in silico systems and expert knowledge for structure-based assessment of potentially mutagenic impurities Regulatory Toxicology and Pharmacology, 2013, 67, 39 Bayer, Sanofi, AZ, Hoffmann-La Roche, Computational Toxicology Services LLC, BMS, Pfizer, Servier, Novartis, J&J, Abbott, Merck, Boehringer, NCSP

17 Enhancing Derek Nexus for mutagenicity Designed to support expert analysis for M7 Provide additional supporting information Recommend where expert should focus analysis If no alerts for mutagenicity were found, Derek Nexus would return Nothing to report With support from our members, we have developed a robust way to extend this and provide further information Next release of Derek Nexus will make an explicit prediction of inactivity for mutagenicity

18 Supporting expert analysis In the absence of a positive alert, experts ask Is there any reason to be concerned with this prediction? Are there any unusual features in my molecule? Are there features associated with false negative predictions? Do I have additional confidential information? a feature = a property derived from structure

19 Negative predictions for mutagenicity Lhasa experts have developed two lists Features known to the model present in the Lhasa Ames Test Ref Set encoded within structural alerts present in Derek examples features not in this list are Unclassified Features found in non-alerting mutagens features present in mutagens that Derek predicts non-mutagenic o These may be coincidental or contributory features in this list are Misclassified Lhasa Ames Test Reference Set contains Vitic, Hansen, FDA, ISSSTY, CGX, Marketed Pharmaceuticals

20 Q Current Derek Nexus Absence of alert returns nothing to report Match alert or example Y Prediction with supporting details N Nothing to report Prediction Likelihood Substructure highlighted Markush Expert comments Validation metrics References Examples

21 Q Next Release of Derek Nexus Assesses the query using the two expert-derived lists Match alert or example Y Prediction with supporting details N Query contains unclassified features? Query contains misclassified features? Prediction Likelihood Substructure highlighted Markush Expert comments Validation metrics References Examples Inactive Inactive with misclassified features Inactive with unclassified features Inactive with unclassified & misclassified features

22 How are compounds classified? Positive predictions are unchanged Nothing to report Inactive Inactive with misclassified features Inactive with unclassified features Inactive with unclassified & misclassified features Partition of public data % % 12 1% 1 - Vitic intermediates % 20 4% 9 2% 0 - Private member data % 29 9% 15 5% 1 - Private member data % 31 7% 13 3% 0 -

23 FN = false negatives : TN = true negatives : Accuracy % How accurate are these classifications? Nothing to report Inactive Inactive with misclassified features Inactive with unclassified features Inactive with unclassified & misclassified features Partition of public data FN 132 / TN % FN 165 / TN 11 6% FN 1 / TN 11 92% FN 1 / TN 0 - Vitic intermediates FN 65 / TN % FN 4 / TN 16 80% FN 3 / TN 6 67% FN 0 / TN 0 - Private member data 1 FN 16 / TN % FN 4 / TN 25 86% FN 0 / TN % FN 0 / TN 1 - Private member data 2 FN 47 / TN % FN 2 / TN 11 86% FN 2 / TN 29 94% FN 0 / TN 1 -

24 Database search / proprietary data could alleviate concerns Example with an unclassified feature No alerts contain this system No examples in the Lhasa Ames Test Reference Set Highlights where to focus to increase confidence inactive

25 Example with a misclassified feature Review examples where the feature was seen in false negative predictions No reason for an expert to over-rule the prediction of Derek Proceed with confidence in a negative prediction

26 Derek Nexus - supporting expert analysis Providing additional direction to focus attention on which features are associated with uncertainty Tests suggest that it is unusual for features to be highlighted. Less than ~10% of the time Accuracy remains high when features are highlighted We are still confident in the prediction

27 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment from 2 predictions Avoiding the problem GTIPurge project

28 Sarah Nexus an advanced statistical system Designed to address the ICH M7 guidelines Created with input from the FDA under a Research Collaboration Agreement Our initial objectives demanded we balance Accuracy Confidence Interpretability Support Applicability Self Organising Hypothesis Networks: A new approach for representing and structuring SAR knowledge. Thierry Hanser et. al. J. Cheminformatics in press

29 Self organising hypothesis networks Local models Global model Model Network

30 Building the Model Curated data (Ames) fragment dictionary Most discriminating patterns Decision tree SOHN (Hypotheses) Pattern refinements

31 Each hypothesis is condensed knowledge Nodes predict fragment activity and give Explanation The fragment Prediction Positive or negative Confidence Strength of signal Supporting examples Ordered by relevance

32 Self organising hypothesis networks

33 Making a prediction Query compounds are fragmented Each fragment is assessed Fragments not covered by the training set result in no prediction Relevant hypotheses for each fragment are retrieved Hypothesis, signal, confidence, supporting examples Typically several hypotheses are returned out of domain Overall Prediction = f (prediction, confidence) hypotheses Absence of a strong overall signal equivocal

34 Confidence correlates with accuracy TN 29% TP 31% FP 22% FN 18% TN 40% FP 13% TP 37% FN 10% TN 39% FP 9% TP 50% FN 2% TN 34% FP 4% TP 60% FN 2% FP 6% TN 23% TP 70% FN 1% 1 b. aaa = ssss + ssss PPP = TT TT + FF 0.6 NNN = TT TT + FF % 20-40% 40-60% 60-80% % Sarah confidence score

35 Sarah s interface prediction confidence hypotheses Structure supporting examples

36 Sarah Nexus regulatory settings Weighted is our recommended option (default) Gives the best model performance Takes into account deactivating features Equivocal Threshold below which signal is not considered strong enough All models balance sensitivity vs specificity Raising sensitivity increases true & false positives Reducing it increases specificity

37 Sarah Nexus analysis of settings Dataset 1 (sub-set Vitic intermediates) o 793 compounds : 30% positive Model settings Model performance metrics Equivocal sensitivity Sensitivity Specificity Coverage Balanced Accuracy 0% 0% % 0% % 10% % -10% Dataset 2 (confidential member) o 513 compounds : 19% positive Model settings Model performance metrics Equivocal sensitivity Sensitivity Specificity Coverage Balanced Accuracy 0% 0% % 0% % 10% % -10%

38 Sarah Nexus Performance Sarah Nexus has been extensively evaluated by members 100% 80% 83-96% 60-85% 60-89% 38-84% 60% Private 1, n= 744, 28% +ive Private 2, n = 847, 12% +ive Private 3, n= 437, 16% +ive 40% 20% 0% sens + spec 2 TN TN + FP TP TP + FN Coverage Balanced accuracy Specificity Sensitivity Private 4, n = 986, 4% +ive Private 5, n = 1718, 14% +ive Private 6, n = 320, 23% +ive FDA, n=809, 36% +ive Public, n = 11209,49% +ive Sarah Nexus v1 under recommended settings Poster 27 th March am, Abstract = 2262 Poster Board = 442

39 Sarah Nexus - Summary Sarah is a statistical approach to mutagenicity Advanced machine-learning approach More effectively captures knowledge Minimizes impact of coincident fragments Maintains high coverage even with challenging datasets Provides information needed for expert analysis Benchmarking Assessment of Open Source and Newly Released Salmonella Mutagenicity (Q)SAR Models for Potential Use Under ICH M7 L. Stavitskaya FDA, poster 2273b Establishing Best Practice for the Application of a Novel Statistical-Based Model to Evaluate Potential Mutagenic Impurities under ICH M7 C. Barber poster 2262 [March 27 th 8.30am]

40 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment from 2 predictions Avoiding the problem GTIPurge project

41 Using in silico predictions M7 explicitly states that in silico predictions should be reviewed with expert knowledge Provide supportive evidence for any prediction Elucidate underlying reasons in case of conflicting results But how will this work in real life? In silico methods combined with expert knowledge rule out mutagenic potential of pharmaceutical impurities: An industry survey Regulatory Toxicology and Pharmacology, 2012, 62, Use of in silico systems and expert knowledge for structure-based assessment of potentially mutagenic impurities Regulatory Toxicology and Pharmacology, 2013, 67, 39

42 2 complementary methodologies should be applied Data methodology Expert system uses all Lhasa data including consortia & donated confidential data + data mined on-site expert system human-written rules based upon data & knowledge Statistical system only uses non-confidential data statistical model machine-learning model using a hierarchical network scope of alert hand-written Markush fragments learnt by model interpretability references expert commentary mechanistic explanation scope of alert some supporting examples transparent methodology learning summarised by hypothesis direct link to training set confidence in prediction

43 Using Sarah and Derek together How often do they disagree? When they agree, how accurate are they? 100% 69-85% 62-90% 80% 60% 40% 20% Private Dataset 1 Private Dataset 2 Private Dataset 3 Public Dataset 0% Agreement between Derek Nexus and Sarah Nexus Balanced accuracy for concurring predictions Acknowledgements : All the Lhasa members who worked closely with us during the evaluation and development of Sarah

44 Using Sarah and Derek together A simple conservative approach will increase sensitivity sensitivity 1..but at the cost of accuracy and specificity , = Private dataset accuracy specificity

45 Using Sarah and Derek together When they disagree, which is right? Public Dataset Private Dataset 3 7% 14% 7% 7% 11% 9% 5% 7% 9% 72% 73% 80% 31% 25% 17% 27%

46 Handling conflicting predictions Confidence scores can give an indication Machine-learnt & expert driven rules have been assessed If both models agree Take that consensus prediction If one model has a high confidence prediction Take the most confident prediction If Derek says positive and Sarah has a positive hypothesis (despite being negative overall) Activity is most likely If the positive prediction is of low confidence Activity is unlikely. Poster 27 th March am, Abstract = 2262 Poster Board = 442

47 Handling conflicting predictions Private Dataset 1 Step Step 2 Step 3 Step 4 D and S agree Most confident prediction D says positive, S has positive hypothesis Low confidence positive Accuracy Sensitivity True accuracy Coverage Simple rules give increased coverage without loss of accuracy Poster 27 th March am, Abstract = 2262 Poster Board = 442

48 Ultimately, expert review is needed Decision trees may help guide an expert, but expert review is still essential We have worked with our members to deliver the information needed for expert review

49 Supporting the expert workflow Step 1 Derek search Predicts inactive but highlights a ring system to assess.

50 Supporting the expert workflow Step 2 Vitic search similarity chosen Vitic shows a related active for which there is no obvious cause (no Derek alert fires) and also a related inactive Expert assessment ring system not of concern

51 Supporting the expert workflow Step 3 Sarah prediction Sarah predicts inactive; no positive hypotheses seen Derek and Sarah analysis agree Supporting data from Vitic augments this prediction

52 Possible reasons to over-rule a positive in silico call The presence of a second confounding alert that could have caused the activity a risk with statistical models Minimised with Sarah s recursive learning approach Mechanistic interpretation stereo-electronics preclude reaction through the accepted mechanism such as that described within Derek Similar analogues trigger the same alert and have been tested as inactive were not known to the model

53 What our members say Combined use of two complementary in silico systems such as Derek Nexus and SEP leads to an increase in negative predictivity and sensitivity, up to 99.1% and 94.7% respectively Poster Comparative Evaluation of in Silico Systems for Ames Test Mutagenicity Prediction Ilse Koijen Janssen, GTA Newark Oct 2013, SEP = the pre-release version of Sarah

54 The use of integrated in silico solutions under the proposed ICH M7 guidelines OUTLINE Impact of changes driven by M7 In silico solutions Vitic Nexus an authoritative toxicity database Derek Nexus the leading expert system Sarah Nexus an advanced statistical system Expert assessment of 2 predictions Avoiding the problem GTIPurge project

55 A Collaborative GTI Purge Estimator ICH M7 Step 2 draft guideline. Understanding.. process.. and impact on residual impurity levels.. with sufficient confidence that the level of the impurity in the drug substance will be below the acceptable limit such that no analytical testing is needed... P.11 Section 8.1 Control of Process Related Impurities, Option 4 GTI = genotoxic impurity

56 The principle is widely accepted Publications describing approaches Strategies for the Evaluation of Genotoxic Impurity Risk Chapter 9 Genotoxic Impurities, Wiley AZ Risk Assessment of Genotoxic Impurities in New Chemical Entities : Strategies To Demonstrate Control Org. Process Res. Dev., 2013, 17 (2), pp 221 AZ, Abbott, GSK, Amgen, Takeda The Assessment of Impurities for Genotoxic Potential and Subsequent Control in Drug Substance and Drug Product J Pharm Sci, 2013, 102, 1014 Lilly

57 A Collaborative GTI Purge Estimator Can Lhasa help produce a tool? Define and automate a best practise process Engage with regulators Coordinate a data-sharing consortium provide support for purge estimate Support regulatory submissions

58 A Collaborative GTI Purge Estimator Create a knowledge base Collect experimentally observed purge factors Remove confidential details Resource for consortium Create a tool that matches the workflow Inputs Process Outputs

59 A Collaborative GTI Purge Estimator Desired outcomes Agreed best practise process captured by software Knowledge base of facts to support the assessment Presentation of the facts in a way that supports regulatory acceptance Support route development to reduce the potential for impurity in final product Has the potential to offer significant cost and time savings Reduce the need for analytical methods to confirm the absence of a potential impurity.

60 Summary M7 will allow predictions of mutagenicity to be submitted Derek has been extended to increase support for expert review Making confident predictions of inactivity Highlighting features worthy of attention Sarah has been designed to provide the statistical 2 nd system Recursive learning and a hierarchical network provide transparency and accuracy The performance of combined predictions has been described Using a number of relevant confidential datasets Examples of expert decision-making illustrate their application Use of Vitic, an authoritative database supports this workflow The collaborative GTI Purge project will support M7 Evidence-based arguments for why an impurity will be lost in synthesis

61 Questions? Government / Regulator Speciality Chemicals Tobacco Veterinary Academic Agrochemica l Lhasa stand 656 Pharma Biotech Pharma Chemical Petrochemic als Personal Products Multiple Medical Devices Generics Food and Nutrition Consultant /CRO Cosmetic Flavours and Fragrances

62

63 Confidence e 1 (+) e 2 (-) e 3 (+) w 1 w 2 w 3 Local model (includes the hypothetical toxicophore)

64 Our assessment of performance true accuracy A non-prediction is a wrong prediction True accuracy = ccccccc ppppppppppp aaa ppppppppppp + nnn ppppppppppp

65 A word on applicability domains Applicability domains response & chemical structure space in which the model makes predictions with a given reliability - Netzeva, Alternatives to Laboratory Animals 2005, 33, 155 Comparison of different approaches to define the applicability domain of QSAR models. - Sahigara, Molecules 2012, 17(5), 4791 is my compound in the model s applicability domain? am I confident enough in this prediction to decide?

66 Sarah Nexus recommended settings Depends upon the use-case: Settings Equivocal Sensitivity Recommended use Out-of-the-box 0% 0% Maximum coverage Accuracy 10% 0% Regulatory 10% 10% Reduced coverage to focus on more accuracy Don t make unconfident predictions Correctly predict more positives Screening 10% -10% Don t make unconfident predictions Correctly predict more negatives

67

68 Example with an unclassified feature

69 Supporting the expert analysis

70 Using Sarah and Derek together Public dataset sensitivity,, 0.7, , specificity

71 Performance of different ways of combining Derek and Sarah predictions Public Dataset Private Dataset Accuracy Sensitivity True Accuracy Derek Sarah S+ or D+ (conservative) S+ & D+ (agreement) Accuracy Sensitivity True Accuracy Derek Sarah S+ or D+ (conservative) S+ & D+ (agreement) Private Dataset 2 Private Dataset Accuracy Sensitivity True Accuracy Derek Sarah S+ or D+ (conservative) S+ & D+ (agreement) Accuracy Sensitivity True Accuracy Derek Sarah S+ or D+ (conservative) S+ & D+ (agreement)

72 Sarah Nexus Progress Jan 2012 Dec 2012 March 2013 April 2013 June 2013 Sept 2013 Oct 2013 Dec 2013 March 2014 Research group identify best methodology Work to produce a Proof of concept starts Dr Hanser describes science behind Sarah at the Japanese ICGM Sarah Engine Prototype (SEP) demonstrated at SOT Member evaluations start First feedback from a regulator obtained Decision to produce Sarah First public lecture describing Sarah made (GTI conference, Berlin) 2 virtual ICGM s giving details to members Sarah announcement reaches the Wall Street Journal Sarah launched First member poster describing SEP s performance FDA present poster using SEP to build strain-specific mutagenicity model First member sponsorships start Publications describing the detailed science submitted Posters presented at SoT2014

73 Supporting the expert workflow Derek predicts inactive Sarah predicts active.but very low confidence

74 Supporting the expert workflow Looking at the training set for Sarah Some closely related compounds are active but could have all been active because also polycyclic aromatics

75 Supporting the expert workflow The Derek alert for alkylating agents didn t fire This fragment is specifically excluded in A27 Conclusion Derek says inactive Sarah weakly active but evaluation of the training set shows another reason for their activity exploration of the Derek alert for alkylating agents (A027) specifically excludes this fragment Expert overall call = Inactive