Setting Standards and Raising Quality for Clinical Bioinformatics. Joo Wook Ahn, Guy s & St Thomas 04/07/ ACGS summer scientific meeting

Transcription

1 Setting Standards and Raising Quality for Clinical Bioinformatics Joo Wook Ahn, Guy s & St Thomas 04/07/ ACGS summer scientific meeting

2 1. Best Practice Guidelines Draft guidelines circulated to labs for comment

3 Best practice guidelines Draft guidelines Code review is a requirement - Necessity of code review was questioned due to resource concerns - Bioinformaticians would like to ensure that these concerns are fully informed

4 Code review Actually a spectrum of activities. The purpose of code review is to deliver a high quality diagnostic service.

5 Code review is peer review

6 Code review can be light

7 Without code review One person - their competency (to write & edit code) is unassessed - their work is unassessed - all NGS testing depends on their work - no one understands their work - if they leave, it s not fun

8 Without code review One person - their competency (to write & edit code) is unassessed - their work is unassessed - all NGS testing depends on their work - no one understands their work - if they leave, it s not fun This is likely to be unaccreditable

9 Code review is required A clinical validation will not detect all potential errors if an NGS pipeline E.g. User interfaces, parts of code that are only applicable in specific situations, probablistic or quantitative results Today s pipelines are very simplistic but rapidly becoming complex

10 Implementing code review Code review is a spectrum of activities: Pair-programming, i.e. two people do everything Change review, i.e. change requests reviewed, as well as how they were implemented (similar to lab SOP review) High-level logic, i.e. high-level discussion of what the code does, e.g. takes aligned sequence data and counts the number of reads at each base - should we do this before or after the sequence quality filter is applied?

11 Implementing code review Code review in the form of high-level logic is already happening (unless the bioinformatician is truly working in isolation) Guidelines do not specify the type of code review to allow for differing resource availability

12 Code review is required Bioinformaticians feel code review is a requirement, not a recommendation, and can be achieved without significant additional resources

13 2. Recidivism Reoffending

14 137 questions -> risk of recidivism score

15

16 Validation results Risk of recidivism score Accuracy = 68% (n=2,328 people) Slightly less predictive for black men than white men 67 percent versus 69 percent.

17 Low score indicates low risk of recidivism Source: ProPublica analysis of data from Broward County, Fla.

18 AFRICAN AMERICAN MEN Low score indicates low risk of recidivism WHITE MEN Source: ProPublica analysis of data from Broward County, Fla.

19 Overall, the tool correctly predicts recidivism of the time, but... Source: ProPublica analysis of data from Broward County, Fla.

20 For best practice, (at least high-level) peer review of code is required.

21 3. Benchmarking exomes

22 Testing genes for which variant detection has not been validated Clinical / focused exomes Wholes exomes Whole genomes

23 Testing genes for which variant detection has not been validated 1. Can variants be detected? 2. Are all genes covered?

24 SNVs insertions <50bp / bp / >200bp deletions <50bp / bp / >200bp complex variants (SNVs and indels within 50bp of each other) inversions tandem duplications copy number gain/dispersed duplications

25 SNVs Sequence context insertions <50bp / bp / >200bp deletions <50bp / bp / >200bp complex variants (SNVs and indels within 50bp of each other) inversions tandem duplications e.g. first exons, repeats Coverage context copy number gain/dispersed duplications Phase

26 Source: Justin Zook, GA4GH

27 VCF-I Truth VCF Query VCF Comparison Engine vcfeval / vgraph / xcmp / bcftools /... Confident Call Regions Two-column VCF with TP/FP/FN annotations VCF-R Two-column VCF with TP/FP/FN/UNK annotations Quantification e.g. quantify / hap.py Counts Stratification BED files Source: Peter Krusche;

28 First step Aim: standardised benchmarking ACGS benchmarking data for SNVs (NA12878/GIAB) ACGS benchmarking data for indels (NA12878/GIAB) + ACGS truth set

29 First data Clinical (reported; Sanger) indels selected Long range PCR Shear MiSeq Very high read depth (thousands) data n = 139 FASTQ & truth VCF generated Sent to all NHS genetics bioinformaticians

30 First results Variant caller GATK-lite (unified genotyper) Samtools 7 NextGene GATK v3 2 GATK v3 1 Platypus 0 GATK v3 0 9 Number of indels missed (n=139) 3 (+ 24 mis-annotated)

31 First conclusions GATK v3 & Platypus are good for detecting indels in high read depth data Different implementations can affect results Normalising VCFs is difficult

32 Next steps Repeat at read depths representative of Panel tests Exome tests Need a standardised benchmarking method to enable apples to apples

33 Pavlos Antoniou Austin Diamond Garan Jones Kim Brugger Kevin Ryan NHS-Bioinformatics Thanks! Michael Yau Oxford University Hospitals NHS Foundation Trust NHS Lothian Royal Devon and Exeter NHS Foundation Trust Cambridge University Hospitals NHS Foundation Trust Guy s and St Thomas NHS Foundation Trust