DATA FORMATS AND QUALITY CONTROL

Transcription

1 HTS Summer School 12-16th September 2016 DATA FORMATS AND QUALITY CONTROL Romina Petersen, University of Cambridge Luigi Grassi, University of Cambridge

2 HTS analysis workflow 2 Biological Sample Prepare library, sequence FASTQ Align short reads to genome VCF, BED, Downstream analysis SAM/BAM Call variants, differential expression,

3 From the sequencer to you 3 Sequencing is usually done by core facilities Each sequencing run will generate millions of short (~100 bp) reads + read quality score for each base They often perform initial processing Adaptor trimming Basic quality control Demultiplexing You will (usually) receive a FASTQ file

4 FASTQ Files 4 FASTQ = FASTA + Quality So what is FASTA?

5 FASTA Format (.fa,.fasta) 5 >CHROMOSOME_1 GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTC AACTCACAGTTTGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATT >CHROMOSOME_2 TCACAGTTTGGTTCAAAGCAGTATCGATCATATCGATCAAATAGTAAA Format for raw DNA/protein sequences For each sequence: 1. > NAME 2. Nucleotides, with line breaks every ~60 bp

6 FASTQ format (.fq,.fastq) 6 Format for DNA sequencing reads For each read: Read ID 2. Nucleotide sequence of the read Quality score for each nucleotide of the read

7 7 Illumina sequence identifiers

8 Quality Scores 8 p = probability that the base call is wrong p = 0.1 à Q = 10 p = 0.01 à Q = 20 P = à Q = 30 Encoding: The Sanger Phred format can encode a quality score from 0 to 93 using ASCII 33 to 126:

9 Quality Score Encoding 9 ASCII Code Quality Score!"#$%&'()*+,-./ :;<=>?@ABCDEFGHI Each character has an associated ASCII Code Quality Score = ASCII Code Offset Normal Sanger Encoding is Phred + 33 Lowest:! = ASCII 33 = Quality 0 Highest: I = ASCII 73 = Quality 40

10 Quality Encoding Example

11 Different Quality Encodings 11 Beware of different versions! (especially for old data)

12 Paired-end reads 12 Sequence both the 5' and 3' end of a fragment Will result in two FASTQ files for a single run file1 n file2 n

13 single-end vs paired-end reads 13 AAGGAAGTGCTTGTCTGGCTAACACAGCNAGNCACGTGAC + avfbe`^^^_tttssdffffdfffabbzbbfebafbbbbb SE NAAATTTCGAATTTCTGTGAAGTAAGCATCTTCTTTGTCAT + BJJGGKIINN^^^^^QQNTUQOOTTTRTOTY^^Y^\\^^^\ my_sequence_2.fastq PE (from the Illumina AACCCACACAGGAGAGCAGCCTTACAGATGCAAATACTGTG + ]K fffffggghgeggggggdgggggfgggggegggghh

14 Sample barcoding and de-multiplexing 14 Barcoding De-multiplexing Barcode 1 Barcode 2 Barcode 3 + Sample 1 Sample 2 Sample 3 # Read 1 ATTAGACCTAAGCA # Read 2 GAGCAACGACTAC # Read 3 ATTAGGCCATACAT # Read 4 CCATAGGCTGACTA...

15 Common sequence artefacts in HTS data 15 q Read errors q Base calling errors q Small insertions and deletions q Poor quality reads q Primer / adapter contamination

16 16 Features across available QC tools

17 17 FastqScreen: contamination

18 FastQC: Per base sequence quality 18 FastQC:

19 19 Quality trimming

20 Removal of adapter sequences 20 q Necessary when the read length > molecule sequenced e.g. small RNAs q Different scenarios requiring adapter removal q q q Trim the 3 end Trim/discard the reads based on the residual minimum read length. Trim the adapter region but retain reads only with a minimum read-length. q Tools for adapter trimming q fastx_clipper (FastX-Toolkit), PRINSEQ

21 How to filter? 21 Fastx: PRINSEQ: Tally and Reaper: ShortRead (R): ShortRead.html

22 FASTQ Processing FASTX Toolkit 22 Many tools for common operations on FASTQ files: Conversion Trimming (remove barcodes) Clipping (remove adapters) Quality trimmer (trim off low-quality bases) Quality filter (remove low-quality reads)

23 Filtering example 23 Before After

24 24 Filtering comes at a price

25 Important: PE files 25 NAAATTTCGAATTTCTGTGAAGTAAGCATCTTCTTTGTCAT + BJJGGKIINN^^^^^QQNTUQOOTTTRTOTY^^Y^\\^^^\ AACCCACACAGGAGAGCAGCCTTACAGATGCAAATACTGTG + ]K fffffggghgeggggggdgggggfgggggegggghh filtering filtering mapping

26 Important: PE files 26 NAAATTTCGAATTTCTGTGAAGTAAGCATCTTCTTTGTCAT + BJJGGKIINN^^^^^QQNTUQOOTTTRTOTY^^Y^\\^^^\ AACCCACACAGGAGAGCAGCCTTACAGATGCAAATACTGTG + ]K fffffggghgeggggggdgggggfgggggegggghh filtering filtering mapping How? Trim Galore! (

27 Conclusions 27 Quality control of sequencing data is essential for downstream analysis. A range of QC tools are available to remove noise. Decide on which data can be corrected and discard the rest.