Alexander Statnikov, Ph.D.

Transcription

1 Alexander Statnikov, Ph.D. Director, Computational Causal Discovery Laboratory Benchmarking Director, Best Practices Integrative Informatics Consultation Service Assistant Professor, Department of Medicine, Division of Translational Medicine Center for Health Informatics and Bioinformatics, NYU School of Medicine 04/11/2011

2 Co-authors: Evgeny Shmelkov (First Author) Zuojian Tang Iannis Aifantis Other collaborators: Constantin F. Aliferis Alexander V. Alekseyenko Timothy Cardozo Yuval Kluger Sergey V. Shmelkov NYU Clinical and Translational Science Institute (CTSI)

3 Core missions: Advise researchers and undertake all aspects of informatics research design and study execution; Review and synthesize of the literature and conducting large-scale benchmarking of a wide array of methods in order to base advice to researchers on solid evidence; Efficiently connect methods developers with methods consumers.

4 Brings experts together; Provides integrated approach to problem-solving. Survey Research Needs Research Problems Impetus for New Methods Best-Practice Recommendations & Methods New Methods Planning Information & User Feedback Planning Information & User Feedback Best-Practice Recommendations & Methods Best-Practice Recommendations & Methods Seek Solutions Implement/ Port & Benchmark

5 There are endless varieties of methods/tools (especially in bioinformatics) and everybody has his/her own preferences BPIC conducts systematic benchmarking of methods/tools in a variety of datasets in order to significantly improve qualify of research

6 Evaluation of classification algorithms for development of molecular signatures from microarray gene expression data Ø Ø Statnikov, et al., Bioinformatics 2005 (~330 citations!) Statnikov, et al., BMC Bioinformatics 2008 (~60 citations!) Evaluation of feature/variable selection & causal discovery algorithms Ø Aliferis, et al., Journal of Machine Learning Research, 2010 Comparison of protocols to detect predictive signal of prognostic molecular signatures Ø Aliferis, et al., PLoS ONE, 2009 Comparison of algorithms for extraction of all maximally predictive and non-redundant molecular signatures Ø Statnikov, et al., PLoS Computation Biology, 2010 (Cover Article) Comprehensive assessment of methods for reverse-engineering of gene regulatory networks Ø Narendra, et al., Genomics, 2011 Assessing quality and completeness of human transcriptional regulatory pathways on a genome-wide scale Ø Shmelkov, et al., Biology Direct, 2011

7 ü As new methods are introduced to the field, we work on revisiting results of our prior evaluation studies. ü New benchmarking efforts: Evaluation of feature/variable selection algorithms for genomics data Evaluation of algorithms for extraction of all maximally predictive and non-redundant feature sets for various application domains Evaluation of methods for next-generation causal orientation from observational data for genomics domains

8 Ø Store huge amount of complex biological data Ø Allow easy access to the collected data representing it in a simple and obvious way of intuitive graphical schemes and notation biological pathways Ø Often provide a variety of integrated tools for rational data handling and analysis by user

9 Ø Basic science hypothesis generation and validation Ø Pharmacology and drug discovery Ø Governmental regulatory agencies such as FDA (!!!)

10 Freely Available: KEGG Biocarta Cell Signaling Technology Pathways WikiPathways Commercial: Pathway Studio Ingenuity Pathway Analysis MetaCore Biobase TRANSPATH Biobase TRANSFAC GeneSpring Pathways

11 Ø Publicly available vs. commercial Ø Different ways of data notation and representation Ø Different approaches to data collection and handling (e.g. expert curation, computational datamining of biomedical literature) à Often proprietary and not fully disclosed for commercial products

12 Shmelkov et al. Biology Direct (2011)

13 Jaccard Index (J) - normalized measure of similarity between two gene lists (A and B): Shmelkov et al. Biology Direct (2011)

14 Data from different pathway databases/ tools are different What pathway database/tool is more reliable?

15 Ø User-friendly interface Ø Colleague advices Ø Prior experience Ø Marketing presentations Ø Software availability and cost To date there have been no prior genome-wide evaluation studies assessing pathway databases and tools!

16 Assess the accuracy and completeness of available data on transcriptional regulation stored in popular pathway databases and tools

17 1. Collect highly-reliable genome-wide experimental data for well-studied transcription factors 2. Integrate the data in order to create reliable lists of direct targets of these TFs 3. Compare by statistical means obtained Gold Standards with relevant lists of transcriptional targets derived from 10 commonly used pathway databases

18 1. Select a TF 2. Obtain a list of all genes bound by the TF using data from ChIPchip or ChIP-seq studies (L1) 3. Using gene expression data (e.g. Microarray or RNA-seq data) obtain a list of genes that are differentially expressed in cell lines that have wild-type TF versus knocked-out TF (L2) 4. Obtain a Gold Standard for the TF by integrating L1 and L2 Jothi et al. Nucleic Acids Research (2008)

19 Ø L1 contains all genes that bind to the TF However, not all genes in this list are functionally regulated by the TF! Ø L2 contains all genes downstream of the TF that are functionally regulated by it However, this list contains both direct and indirect downstream targets of the TF! L1: All genes that are bound by the TF L2: All downstream targets of the TF (functionally regulated) Ø Thus, we need to intersect these two lists L1 and L2 to get a Gold Standard

20 TF Primary activity Gold Standard Number of Genes in the Gold Standard Analyzed to find genes that are AR BCL6 MYC NOTCH1 RELA STAT1 TP53 N/A Repressor Activator Activator Activator Activator Activator I 513 differentially expressed (experiment treated for 4 hr) II 712 differentially expressed (experiment treated for 16 hr) III 526 differentially expressed (all treated) I 369 differentially expressed II 98 differentially expressed III 271 up-regulated in treated samples IV 76 up-regulated in treated samples I 1887 differentially expressed II 1708 down-regulated in treated samples III 3039 differentially expressed IV 2224 up-regulated in MYC-expressing samples I 414 differentially expressed II 302 down-regulated in treated samples III 637 differentially expressed IV 471 down-regulated in treated samples V 167 differentially expressed VI 166 down-regulated in treated samples I 1864 differentially expressed II 1420 down-regulated in treated samples III 188 differentially expressed IV 136 up-regulated in treated samples I 2128 differentially expressed II 2967 down-regulated in treated samples I 34 differentially expressed II 37 down-regulated in treated samples Shmelkov et al. Biology Direct (2011)

21 Ø Microarrays cannot reliably detect small changes in gene expression and/or genes expressed on very low levels Ø ChIP-chip and ChIP-seq transcription factor-dna binding data is known to have imperfect reproducibility Ø Functional gene expression and binding data used in our work often originated from different studies Ø Compensatory mechanisms in the cell can cause some number of false negatives

22 Ø Ø Ø Ø The statistical significance of the overlap between Gold Standards and lists from Pathway Databases/ Tools is calculated using the hypergeometric test The Null-hypothesis of the test is that two assessed sets of genes have not greater number of genes in common than two randomly selected gene sets with the same number of genes The Null-hypothesis is rejected at 5% α-level False discovery rate (FDR) correction was applied for multiple comparisons analysis Shmelkov et al. Biology Direct (2011)

23 Shmelkov et al. Biology Direct (2011)

24 Even though hypergeometric test is based on odds ratios, databases with a very small number of targets may not reach statistical significance regardless of the quality of their data. Enrichment Fold Change (EFC) ratios are calculated as the observed number of genes in the intersection divided by the expected size of intersection under the null hypothesis Notice however that larger values of EFC may correspond to databases that are highly incomplete and contain only a few relations. Shmelkov et al. Biology Direct (2011)

25 Shmelkov et al. Biology Direct (2011)

26 The lists of experimentally derived direct targets obtained in this study can be used to reveal new biological insight in transcriptional regulation and suggest novel putative therapeutic targets. We and others have previously suggested that induction and maintenance of T-cell acute lymphoblastic leukemia (T-ALL), a devastating pediatric blood cancer, depends on the cross talk of three transcription factors, NOTCH1, MYC, and RELA (NF-κB). NOTCH1 & MYC è 438 common targets Two activators of cell cycle entry, CDK4 and CDK6 appear to be induced by both factors. Aifantis et al. has previously shown that silencing of CDK4/6 activity is able to suppress T-ALL suggesting that NOTCH1 and MYC activities could converge on these CDK genes to initiate expansion of transformed cells. MYC itself and its interacting partners MYCB and MYCB2 appear to also be targeted by both factors, suggesting an interesting signal amplification mechanism.

27 RELA & MYC è 561 common targets Several essential T-cell regulators, including RUNX1, BCL2L1 (BCL-xL), ID3, ITCH, JAK3 and NOTCH1, appear to be controlled by both transcription factors. NOTCH1 is downstream of both RELA and MYC but at the same time these two factors are targets of oncogenic NOTCH1, suggesting once more an intricate auto-amplification loop that could sustain transformation. NOTCH1 & RELA è 156 common targets NOTCH1 & RELA & MYC è 117 common targets (Lists of genes are given in the paper supplement)

28 Ø Pathway Databases/Tools often do not agree with each other and contain different target genes of assessed transcription factors Ø Majority of sets of target genes extracted from selected Databases/Tools are most likely incomplete and/or inaccurate as they significantly disagree with experimentally derived Gold Standards (MetaCore is the only exception)

29 Ø In order to obtain a more accurate research hypothesis, the choice of pathway databases has to be informed by solid scientific evidence and rigorous empirical evaluations such as ours Ø We recommend developers of pathway software to Take advantage of high-throughput genome-wide data to refine pathways instead of traditional literature search for single interactions Provide detailed description of each interaction and node in the pathway (e.g. introduce confidence scores reflecting the reliability level of original reference data; address differences in experimental conditions)

30 Assessing quality and completeness of human transcriptional regulatory pathways on a genome-wide scale Evgeny Shmelkov, Zuojian Tang, Iannis Aifantis, Alexander Statnikov Biology Direct (2011), 6:15 6/1/15 Gold Standards could be accessed at:

31 Alexander Statnikov: Evgeny Shmelkov: