Statistical method for Next Generation Sequencing pipeline comparison

Transcription

1 Statistical method for Next Generation Sequencing pipeline comparison Pascal Roy, MD PhD EPICLIN 2016 Strasbourg mai 2016 MH Elsensohn 1-4*, N Leblay 1-4, S Dimassi 5,6, A Campan-Fournier 5,6, A Labalme 5, D Sanlaville 5,6, G Lesca 5,6, C Bardel 1-4, P Roy Service de Biostatistique, Hospices Civils de Lyon, Lyon, France. 2 Université de Lyon, Lyon, France. 3 Université Lyon 1, Villeurbanne, France. 4 CNRS UMR 5558, Laboratoire de Biométrie et Biologie Evolutive, Equipe Biostatistique-Santé, Villeurbanne, France. 5 Service de Génétique, Hospices Civils de Lyon, Lyon, France 6 Centre de recherche en Neurosciences de Lyon (CNRS UMR 5292, INSERM U1028, Université Claude Bernard Lyon 1, Université Jean Monnet Saint-Etienne, and Hospices Civils de Lyon) Lyon, France. 1

2 DNA sequencing Sanger method (1977) considered as a Gold Standard New sequencers were designed since 2000 Next Generation Sequencing technologies became available since

3 Pipeline Association of software programs for the various steps of NGS data analyses Academic and commercial softwares are available How To compare two NGS pipelines once with another? Each with the Gold Standard Sanger? 3

4 DNA Sequencing NGS sequencing Sanger technique Genetics Department - Hospices Civils de Lyon Whole blood Informed consent 43 epileptic patients 41 genes (epilepsy / mental retardation) Ion Torrent PGM BWA-GATK and TMAP-NextGENe pipelines 30 epileptic patients 1 to 3 genes according to clinical signs 4

5 Statistical Unit = Chromosomal position on the reference sequence Hg19 Each patient = A single study All patients = A meta-analysis 5

6 2-by-2 pairwise table for agreement p = 1,, P patient z = A, B Pipeline k = 1,, K X pzk nπ pab 1 0 K k 1 otherwise Chromosomal position disagreeme ntwithhg19 I X a I X b,a 0,1,b 0,1 pak pbk 6

7 Agreement on position Agreement on position and nature 7

8 With Gold Standard Sanger variants Sensitivity comparison Sanger non-variants Specificity comparison 8

9 9

10 1 2 P-1 P 10

11 1 2 Saturated model for variant identity A B log(npab )= μp +aλp +bλp +ab(θ p +Iθ ps Alternative approach Fitting mixed-effects log-linear models Random effects for biological variability ) P-1 P 11

12 # of Variants identified by the pipelines and by the gold standard All Types of variants* Variants and pipelines N Mean±SD Min. Max. Regions sequenced by MPS only BWA-GATK ± TMAP-NextGENe ± Region sequenced by MPS +Sanger Sanger ± BWA-GATK ± TMAP-NextGENe ± * Single Nucleotide Variants (SNVs), insertions, and deletions base-pairs per patient 1 to 3 genes and to base-pairs per patient 12

13 # of Variants identified by the pipelines and by the gold standard Only SNVs Variants and pipelines N Mean±SD Min. Max. Regions sequenced by MPS only BWA-GATK ± TMAP-NextGENe ± Region sequenced by MPS +Sanger Sanger ± BWA-GATK ± TMAP-NextGENe ± base-pairs per patient to base-pairs per patient 13

14 Pipeline comparisons - All types of variants Estimation of parameters Variants, pipelines, parameters Value 95% CI 95% BVI Without Gold Standard * # of variants for BWA-GATK ; ; # of variants for TMAP-NextGENe ; ; OR for agreement ; ; Conditional probability of identity ** ; ; 0.28 With Gold Standard Sensitivity of BWA-GATK (%) ; ; Sensitivity of TMAP-NextGENe (%) ; ; FP rate for BWA-GATK ; NA FP rate for TMAP-NextGENe ; NA * Heterogeneous margins and odds-ratios. **Heterogeneous variant identity parameter Heterogeneous margins and odds-ratios for specificity extended to sensitivity analysis for Sanger NV 14

15 Pipeline comparisons - Only SNVs Estimation of parameters Variants, pipelines, parameters Value 95% CI 95% BVI Without Gold Standard * Number of SNVs for BWA-GATK ; ; Number of SNVs for TMAP-NextGENe ; ; OR for agreement ; ; Conditional probability of identity ** ; NA With Gold Standard Sensitivity of BWA-GATK (%) ; NA Sensitivity of TMAP-NextGENe (%) ; NA FP rate for BWA-GATK ; 2.24 NA FP rate for TMAP-NextGENe ; 2.24 NA * Heterogeneous margins and odds-ratios. **Homogeneous variant identity parameter Homogeneous margins and odds-ratios for specificity extended to sensitivity analysis 15 for Sanger NV

16 Discussion Several usual statistical methods may be adapted to NGS analyses More sophisticated experimental designs are needed to analyze the various components of experimental variability, in contrast with biological variability More sophisticated decision rules are needed to select appropriate pipelines, including the respective costs of FP and FN When all variants were analyzed, the performances of the 2 pipelines were low in terms of false sensitivity et specificity In substitution analyses, a better performance was observed, leading to a very high value of the odds-ratio for agreement Extensions of these models are needed to evaluated the performances of NGS 16

17 References Li H. Exploring single-sample SNP and INDEL calling with wholegenome de novo assembly. Bioinformatics 2012;28: Agresti A. Categorical Data Analysis, 3 rd edition. Wiley, Becker MP., Agresti A. Log-linear modelling of pairwise interobserver agreement on a categorical scale. Stat Med 1992;11: