Quality Filtering of Illumina Sequences. Susan Huse Brown University August 6, 2015

Transcription

1 Quality Filtering of Illumina Sequences Susan Huse Brown University August 6, 2015

2 Illumina FASTQ Files File naming: NA10831_ATCACG_L002_R1_001.fastq.gz FA1_S1_L001_R1_001.fastq.gz Sample_Barcode/Index_Lane_Read#_Set#.fastq.gz Sequence A8T0A:1:1101:14740:1627 : Run# : FlowcellID : Lane : Tile : X : Y Read : Filtered : Control# : Barcode/Index

3 @sequence_id sequence + quality FASTQ Format 4 lines per 1:N:0:1 CCTACGGGAGGCAGCAGTGGGGAATATTGCACAATGGGGGAAACCCTGATGC AGCGACGCCGCGTGAGTGAAGAAGTATCTCGGTATGTAAAGCTCTATCAGCA GGAAAGATAATGACGGTACCTGACTAAGAAGCCCCGGCTAACTACGTGCCAG CAGCCGCGGTAATACGTAGGGGGCAAGCGTTATCCGGATTTACTGGGTGTAA AGGGAGCGTAGACGGCAGCGCAAGTCTGGAGTGAAATGCCGGGGCCCAACCC CGGCCCTGCTTTGGAACCCGTCCCGCTCCAGTGCGGGCGGG + 88CCCGDBAF)===CEFFGGGG>GGGGGGCCFGGGGGDFGGGGDCFGGGFED CFG:@CFCGGGGGGG?FFG9FFFGG9ECEFGGGDFGGGFFEFAFAFFEFECE F@4AFD85CFFAA?7+C@FFF<,A?,,,,,,AFFF77BFC,8>,>8D@FFFF G,ACGGGCFG>*57;*6=C58:?<)9?:=:C*;;@C?3977@C7E*;29>/= +2**)75):17)8@EE3>D59>)>).)61)4>(6*+/)@F ??D1 :0)((,((.(.+)(()(-(*-(-((-,,(.(.)),(-0)))

4 Assembly vs. Amplicons Genome Assembly Drops reads that don t match Calculates consensus base at each posison Amplicon and Metagenomics Every read represents an independent copy of the source DNA poor quality sequences become novel organisms or genes

5 Phred Scores Q = -10 * log (p) 1. Take the log of the probability of error 2. Convert to positive integer p Q 0.1 (10%) (1%) (0.1%) (0.001%) 40

6 Theoretical Phred Scores vs Error Probability 1 Probability of Error Phred Scores

7 Theoretical Phred Scores vs Error Probability Probability of Error Phred Scores

8 Fastq Formahed Quality # $ % & ( ) * , -. / : ; < = >? A B C D E F G H I Letters are good

9 Reading FASTQ 1:N:0:1 CCTACGGGAGGCAGCAGTGGGGAATATTGCACAATGGGGGAAACCCTGATGC AGCGACGCCGCGTGAGTGAAGAAGTATCTCGGTATGTAAAGCTCTATCAGCA GGAAAGATAATGACGGTACCTGACTAAGAAGCCCCGGCTAACTACGTGCCAG CAGCCGCGGTAATACGTAGGGGGCAAGCGTTATCCGGATTTACTGGGTGTAA AGGGAGCGTAGACGGCAGCGCAAGTCTGGAGTGAAATGCCGGGGCCCAACCC CGGCCCTGCTTTGGAACCCGTCCCGCTCCAGTGCGGGCGGG + 88CCCGDBAF)===CEFFGGGG>GGGGGGCCFGGGGGDFGGGGDCFGGGFED CFG:@CFCGGGGGGG?FFG9FFFGG9ECEFGGGDFGGGFFEFAFAFFEFECE F@4AFD85CFFAA?7+C@FFF<,A?,,,,,,AFFF77BFC,8>,>8D@FFFF G,ACGGGCFG>*57;*6=C58:?<)9?:=:C*;;@C?3977@C7E*;29>/= +2**)75):17)8@EE3>D59>)>).)61)4>(6*+/)@F ??D1 1:N:0:1

10 Why filter infrequent errors? Ns Average 454 Error Rate Errors / 400nt Percent of Reads 0 or more 0.40% % % % If we include all reads with or without Ns, we have an overall error rate of 0.4%. If, however, we remove all sequences with Ns, we have an overall error rate of 0.4%. Why bother??

11 It s all in your perspective

12 Low Percentage, but High Errors Ns Average Error Rate Errors / 400nt Percent of Reads % % % % % % % % % % % % Low-quality reads can be interpreted as unique organisms: 2nt = 0.13% * 1 million reads = 1,300 unique organisms

13 Impact of Error Rates # errors = (error rate) * (# bases sequenced) Predicted number of errors increases with sequencing depth at Q30 = * 100,000 bases [Sanger] = 100 bases * 300,000,000 bases [Illumina] = 300 thousand bases

14 Low- Quality Reads and Errant OTUs

15 Errant OTUs If a low- quality sequence is >3% from its source, it can create a new OTU. If the rate of an errant read = 1 in 10 thousand, and we have 1 million reads: * 1,000,000 reads = 100 errant OTUs

16 Errant OTUs Errant OTUs as percent of OTUs decreases with diversity. If we have 100 errant OTUs: Mock community: 100 / 50 OTUs = +200% Diverse community: 100 / 2,000 OTUs = +5%

17 Cleaning Data Denoising improve noisy base calls, remap reads Filtering remove low- quality reads and non- target sequences Trimming prune low- quality ends Chimeras remove chimeric reads AggregaMng combine similar reads or taxa

18 EvaluaSng Error Paherns 1. Sequence known template 2. Align the actual read sequences against the expected sequences 3. Evaluate distribuson of sequencing errors 4. Find correlasons between measurable parameters and error rates 5. Assess the contribuson of each error pahern to the overall error rate.

19 Defining the Error Rate Error Rate = the number of errors per base subsstusons + insersons + delesons + uncalled alignment length Read: Template: AGCNC-ATAACTCTG AGCTCGAC--CAGCT 6/15 = 0.40

20 Illumina quality scores reflect error rates Figure 6 Minoche et al Genome Biology

21 Figure 5 Low Q are low quality, High Q usually high quality Minoche et al Genome Biology

22 DistribuSon of Errors Cumulative Percent of Errors 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 79% of error bases have a quality score <=16 12% of error bases have a quality score >= Quality Score

23 Expected error rates after filtering (adapted from Minoche et al, Table 2) Filter PhiX- GAIIx Error Rate / (% of bases discarded) No filter (0.0%) ChasSty filter, Illumina (signal intensity rasos) (17.8%) Low quality tails (25.8%) Ns (15.6%) C33 (Q<30 for 1/3 of bases in 1 st half) (21.7%) ChF +LQ- tail + N + C (28.9%)

24 Sequences with Ns NTAGCACCAAACATAAATCACCTCACTTAAGTGGCTGGAGACAAATAATCTCTTTAATAACCTGATTCAGCGAAACCAATCCGCGGCATTTAGTAGCGGTA NTAATTACCCCAAAAAGAAAGGTATTAAGGATGAGTGTTCAAGATTGCTGGAGGCCTCCACTATGAAATCGCGTAGAGGCTTTGCTATTCAGCGTTTGATG NGCGCCAATATGAGAAGAGCCATACCGCTGATTCTGCGTTTGCTGATGAACTAAGTCAACCTCAGCACTAACCTTGCGAGTCATTTCTTTGATTTGGTCAT NGTAAAAATGTCTACAGTAGAGTCAATAGCAAGGCCACGACGCAATGGAGAAAGACGGAGAGCGCCAACGGCGTCCATCTCGAAGGAGTCGCCAGCGATAA NTCTATGTGGCTAAATACGTTAACAAAAAGTCAGATATGGACCTTGCTGCTAAAGGTCTAGGAGCTAAAGAATGGAACAACTCACTAAAAACCAAGCTGTC CAGTGGAATAGTCAGGTTAAATTTAATGTGACCGTNTNNNNNAATNNNNNNNNNNNNNNNNNNNNNNNCANNNNNTNGNNNNANNNNNTTGAGTGTGAGGT CGGATTGTTCAGTAACTTGACTCATGATTTCTTACCTATTAGTGGTTNAACANNNNNNNNNNNNNATAGTAATCCACGCTCTTNTAANATGTCAACAAGAG TATGCGCCAAATGCTTACTCAAGCTCAAACGGCTGGTCAGAATTTTACCAATGACCANNNCAAAGAAATGACTCGCAAGGTTAGTGCTGAGGTTGACTTAG TAGAAGTCGTCATTTGGCGAGAAAGCTCAGTCTCAGGAGGAAGCGGAGCAGTCCAAANNNTTTTGAGATGGCAGCAACGGAAACCATAACGAGCATCATCT TGCTGTTGAGTGGTCTCATGACAATAAAGTATGTCNCTGNNTTGAAGNNTNNNNNNNNNNNNNNNCTNATACAATCACGCNCANNNNNAAAAGTGTCGTGT CTACTGCGACTAAAGAGATTCAGTACCTTAACGCTAAAGGTGCTTTGNCTTANNNNNNNNNNNNTGGCGACCCTGTTTTGTATGGCANCTTGCCGCCGCGT CGGCAGAAGCCTGAATGAGCTTAATAGAGGCCAAAGCGGTCTGGAAACGTACGGATTNNNNAGTAACTTGACTCATGATTTCTTACCTATTAGTGGTTGAA GTGATTTATGTTTGGTGCTATTGCTGGCGGTATTGCTTCTGCTCTTGNTGGTNNCNNNNNNNNNAAATTGTTTGGAGGCGGTCAAAANGCCGCCTCCGGTG ATATCAACCACACCAGAAGCAGCATCAGTGACGACATTAGAAATATCCTTTGNAGTNNNNNNNNTATGAGAAGAGCCATACCGCTGATTCTGCGTTTGCTG In this dataset: 68 reads contained at least 1 N, of these: 24 (35%) contain more than 1 N 14 (21%) could not be mapped to PhiX, 7 of those 14 (50%) had only 1 N

25 Paired- End Amplicons A smaller insert size provides sequence overlap Read 1 (forward) Sequence overlap Read 2 (reverse)

26 Complete Overlap (V6) Ensures high-quality reads Does not ensure perfect data Read 1 (forward) TGGTCTTGACATCCACAGAACTTTCCAGAGATGGATTGGTGCCTTCGGGAACTGTGAGAC TGGTCTTGACATCCACAGAACTTTCCAGAGATGGATTGGTGCCTTCGGGAACTGTGAGAC Read 2 (reverse) Eren, AM et al (2013) PLoS ONE 8(6)

27 Comparing Filtering Methods Low Quality Perfect Overlap 975,410 (26%) Bokulich et al 391,993 (11%) Minoche et al 435,925 (12%) Figure 3 Eren et al (2013) PLoS ONE

28 Comparing Filtering Methods High Quality Perfect Overlap 2,707,801 (74%) Bokulich et al 3,291,218 (89%) Minoche et al 3,247,286 (88%) Figure 3 Eren et al (2013) PLoS ONE

29 Imperfect Overlap CorrecSon Requiring perfect overlap removes up to 20-30% of the reads. Can we use quality scores to correct bases in the overlap region rather dropping ensre reads? 1. Align the overlap region 2. Compare bases and quality scores 3. Assign most probable base and correct qual score

30 Edgar and Flyvbjerg (2015) Bioinformatics

31 Imperfect Overlap CorrecSon USEARCH fastq_mergepairs Edgar and Flyvbjerg (2015) Bioinforma8cs PANDASeq Masella et al. (2012) BMC Bioinforma8cs merge- illumina- pairs Eren et al. (2013) PLoS ONE PEAR Zhang et al (2014) Bioinforma8cs

32 Paired Overlap Parameters Matter IniSal Reads 1,536,548 Merge- illumina- pairs (max 3 mismatches) 996,139 à 467,792 uniques PANDASeq (no Ns, 95% similarity) 996,139 à 804,546 uniques

33 Edgar and Flyvbjerg (2015) Bioinformatics

34 Denoising Assume that sequencing errors lead to a stassscal distribuson of reads around the more abundant true error- free sequence. Use the error distribuson to map probable error reads to their probable source sequence. AmpliconNoise (454) - Quince et al (2011) BMC Bioinforma8cs DADA (DADA2) - Rosen et al (2012) BMC Bioinforma8cs

35 What other sources of error should we check for?

36 Chimeras Not an error in sequencing but in amplificason (see Chimeras lecture)

37 Non SSU rrna AmplificaSon Conserved inner membrane protein cardiolipin synthase Predicted major pilin subunit 16S rrna DNA binding transcriptional dual regulator, tyrosinebinding Putative transport system permease protein Predicted antibiotic transporter 16S rrna Courtesy of Hilary Morrison

38 16S From Other Domains SSU region Total Reads Bacteria Archaea Organelle Unknown V6 529,359 96% 0.02% 4% 0.1% V6- V4 3,437,855 87% 0.3% 8% 4% Use taxonomic filtering to remove non-target DNA Samples from Little Sippewissett Marsh. Organelles include mitochondria and chloroplasts

39 Bar Hopping Barcoding Errors can cause reads to hop from one sample to another AGATC = Sample1 AGATT = Sample2 AGATA =??? Always use codes >= 2nt different Always require <=1 mismatches (0=best)

40 AggregaSng Small Errors Taxonomic assignments are generally consistent despite a few mismatches. More so at coarser taxonomic levels (class vs. genus) OTU Clustering and Oligotyping round out small percentages of errors depending on the algorithm used. Clustering at 3% can (but does not always!) aggregate sequences with 1 2% errors.

41 Singleton Errares Singletons can be: valid = rare organisms or invalid = sequencing errors Singletons that pass quality control can only be validated ecologically

42 Singleton Errares Valid singletons represent rare organisms Invalid singletons are sequencing errors Absolute number of errors increases with sampling depth. Errors as percent of uniques decreases with diversity If you choose to remove singletons, only a er filtering, aggregason, and comparison across datasets.

43 Navy minesweeper runs aground, due to faulty charts

44 General Caveats Always maintain a healthy skepscism about the quality of any sequencing data Never underessmate the presence or impact of low- quality data or untargeted DNA Not all infrequent sequences are bad sequences Be vigilant for taxonomically- biased filtering Don t skimp on quality filtering!!!