CASE-STUDY- VALIDATION of PCR based methodology Beata Surmacz-Cordle Senior Analytical Development Scientist UK RMP Pluripotent Stem Cell Platform Validation Workshop 2 nd June 2016
RT-qPCR assay for detection of persistence of genetically modified T cells in autologous cell therapy
Introduction to Case Study RT-qPCR assay Validation Summary
Intro
CASE STUDY Modified T cell receptor (TCR) autologous Cell Therapy Start -1 month Day 0 few years T Cell T Cell T Cell T Cell Genetically engineer to modify specificity T Cell T Cell T Cell T Cell Infuse back into patient Patient monitoring: Multiple Clinical Trial time points Harvest T Cells Genetically modified T Cell Receptor Therapy Anti-Tumour Activity How many TCR modified cells per 1x10E6 WBC? Persistence of TCR modified cells RT-qPCR assay
Persistence of TCR modified cells RT-qPCR assay principle RT-qPCR assay evaluates the persistence of genetically modified T cell receptor cells directed against blood leukaemic cells; Presence of genetically modified TCR expressing cells is analysed using RT-qPCR assay in samples pre and post infusion; RT-qPCR is a clinical assay (Good Clinical Laboratory Practise) that is performed multiple times over few years testing numerous patient blood samples; Detection of RNA or DNA? RT-qPCR assay provides quantitative information about the number of cells that are actively transcribing the modified TCR receptor; Assay is RNA based (RT +qpcr)
Ct Persistence of TCR modified cells RT-qPCR assay principle Standard Curve based readout (Serial dilution of Transduced cells in mock Transduced cells) PBMC Preparation RNA isolation cdna synthesis -RT qpcr assay 10 cells 100 1K 10K 100K X TCR Cell Number SYBR Green based assay with TCR specific and house keeping primers (2 sets of primers)
Validation?
PCR assay workflow to Validation Optimisation (evidence) Qualification (gain confidence and learn about assay limits) Validation (demonstrate that it is suitable for intended purpose!)
Step 1.2 Step 1.1 RT-qPCR assay analysis tools ( paper based ) Project Code: Template100x04 Last Modified: 24/02/2015 Printed: 24/02/2015 Day 0 120 minutes Day 0 10 minutes Cell Product Raw Material Consumable Sample Name the step in bold Put in detail Add subprocesses if required Gap Analysis Analytical Transfer Process Transfer Assay or Donor data Name the step in bold Put in detail Decision Yes No Cell Product Waste Sample Assay or Donor data Rejected Product Final Product Tissue Culture (Grade B) Autoclave (Unclassified) Analytical Lab (Unclassified) Analytical Lab (Unclassified) Equipment Equipment Process Dissection BSD P-Diagram Process Operation Process Parameters C/N/X Quality Attribute 1 Quality Attribute 2 Quality Attribute 3 Quality Attribute 4 Quality Attribute 5 Quality Attribute 6 Quality Attributes (and weightings) Batch Completion Quality Attribute 7 Quality Attribute 8 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Process Operation 1 Input 1 C 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 17 17 Process Operation 1 Input 2 N 1 3 1 1 9 1 3 1 1 1 1 1 1 1 1 1 1 29 29 Process Operation 1 Input 3 X 1 1 9 9 9 1 1 1 Q 1 1 1 1 1 1 1 1 40 40 Quality Attribute 9 Quality Attribute 10 Quality Attribute 11 Quality Attribute 12 Quality Attribute 13 Quality Attribute 14 Quality Attribute 15 Quality Attribute 16 Score FMEA COGS options Weighted Score Risk vs COGS Development Plan Priority Grid Implementation Plan Analysis showed that assay optimisation is required rather validation
Qualification Optimisation RT-qPCR assay Optimisation and Qualification Study 1 : Vicell dynamic range for automated T cell count; Study 2 : Identification of RNA isolation method that gives highest yield; Study 3 : Assessment of correct primer concentration to improve signal to noise ratio without loss of sensitivity; Selection of Assay Pass/Fail criteria: (Noise to Signal ratio= Ct value cut off) Number of technical replicates Appropriate Positive and Negative controls (Transduced and mock Transduced cells, Plasmids for both assays, NTC)
PCR assay workflow to Validation Optimisation (evidence) Qualification (gain confidence and learn about assay limits) Validation (demonstrate that it is suitable for intended purpose!)
RT-qPCR assay validation
ICH, MIQE guidelines and GCLP.. ICH Quideline Q2(R1) 1994 The MIQE Guidelines: Minimum Information for Publication of Quantitative Real-Time PCR Experiments. Clinical Chemistry 55:4 611 622 (2009) qpcr Validation Importance Evidence of optimistion Desirable Specificity (gel, sequence, melt, digest) Essential For SYBR Green, Cq of the NTC Good Essential Clinical Laboratory Practise (GCLP) by RQA, V2 2012 Standard curves with slope and y-intercept Essential PCR efficiency calculated from slope Essential Confidence interval for PCR efficiency or standard error Desirable R 2 of standard curve Linear dynamic range Cq variation at lower limit Confidence intervals throughout range evidence for limit of detection If multiplex, efficiency and LOD of each assay Essential Essential Essential Desirable Essential Essential
Parameters tested for RT-qPCR assay validation Assay Linearity Assay Sensitivity Assay Accuracy Assay Precision Assay Specificity Assay Robustness Acceptance criteria must be well defined before validation for all parameters!
Ct Linearity_1.. the ability to produce test results that are directly, or by a welldefined mathematical transformation, proportional to the concentration of analyte in samples within a given range (ICH Q2, R1) 1 cell 10 cells 100 1K 10K Linearity assessment 4 independently generated standard curves (log dilution of transduced cells); RT-qPCR run on 4 different days; All samples run in multiple replicates; Cell number 100K Acceptance criteria R 2 (1 +/- 0.1) Slope and Efficiency (-3.6 to -3.1; 90-110) (Residual Plot) 1 2 3 4 5 6 7 8 9 10 11 12 A 10 5 10 5 10 5 TD TD TD 10 5 10 5 105 TD TD TD B 10 4 10 4 10 4 mockntd mocktd mocktd 10 4 10 4 104 mockntd mocktd mocktd C 10 3 10 3 10 3 NTC NTC NTC 10 3 10 3 103 NTC NTC NTC D 10 2 10 2 10 2 Control plasmid Control plasmid Control plasmid 10 2 10 2 10 2 Control E 10 10 10 10 10 10 F 1 1 1 1 1 1 plasmid Control plasmid Control plasmid G NoRT TD NoRT TD NoRT TD NoRT TD NoRT TD NoRT TD H NoRT mock Td NoRT mock Td NoRT mock Td NoRT mock Td NoRT mock Td NoRT mock Td
C t(p r e d ic t e d )-C t (m e a s u r e d ) Linearity_2 Results Measurement Desirable value Results Standard Deviation Pass/Fail parameter range between data sets R 2* 1 +/- 0.1 0.993 0.007 PASS Slope -3.6 to -3.1-3.409 0.348 PASS Efficiency (%) 90-110 98.09 9.68 PASS * RT-qPCR assay meets desired criteria over a range of 10 2-10 5 cells Residual Plot 0.5 0.0-0.5 C u rv e 1 C u rv e 2 C u rv e 3 C u rv e 4 0 1 2 3 4 5 6 L o g 1 0 [C e ll n u m b e r ] The RT-qPCR assay fits within the range of acceptable values for assay linearity between 10 2-10 5 cells
C t(p r e d ic t e d )-C t (m e a s u r e d ) Sensitivity_1 the ability of the assay to consistently quantify specimens that contain barely measureable analyte. LOQ Limit of Quantification- the lowest actual amount of an analyte that can be reliably detected and at which the total error meets the laboratory s requirements fro accuracy, so it can be quantified LOD Limit of Detection the lowest amount of analyte in a sample that can be detected with a stated probability of 95% Confidence Intervals 1 cell? 10 cells? 100? 1K 10K 100K Sensitivity assessment 4 independently generated standard curves from 100K to 1 cell (log dilution) 10 replicates! (more might be required) 0.5 0.0 Residual Plot C u rv e 1 C u rv e 2-0.5 C u rv e 3 C u rv e 4 Acceptance criteria 90% positive detection calls (Ct<37) CV%<5 0 1 2 3 4 5 6 L o g 1 0 [C e ll n u m b e r ]
LOQ and LOD are not trivial for PCR assays! Signal statistically different than background (no value readout for background =0) Difficult to apply to PCR assays LOQ may be limited by the linearity of the STD curve (check for residuals plot) LOQ should be within linear range LOQ may be set by the spread of replicates LOQ may never be lower than LOD LOD CV >95% is important LOD in real life depends on: - Extraction yield - Loses during handling, transfer and storage - Losses during dispensing - Reverse Transcription yield - Inhibition and interference in qpcr reaction To determine LOD for full procedure you need to start from the beginning so from RNA extraction..
Sensitivity_2 Results 10 cells 100 1K 10K 100K 10K 1K 100K 100 Measurement parameter Acceptance value LOQ results for TCR specific assay Control results for house keeping assay TCR modified cell number 100 10 100 10 % positive detection calls (Ct <37) 90 97.5 40 100 100 CV (%) <5 2.87 1.44 1.91 1.32 Pass/ fail PASS FAIL PASS PASS LOQ for RT-qPCR assay is at 100 cells
Accuracy is defined as the closeness of the test results obtained by an analytical method to conventional true value or an accepted reference value and the mean value found from replicate or repeated assessment in the assay Accuracy Assessment In this particular assay there is no true value to test for accuracy, therefore accuracy cannot be measured Cell number readout is depended on multiple factors (i.e. flow cytometry assessed transduction efficiency as well as viable cell count and pipetting..)
Precision_1 the closeness of agreement (degree of scatter) between series of measurements obtained from multiple sampling of the same homogenous samples. Precision is solely related to the random error and has no relation to accuracy. An assay can be precise but also inaccurate.! Precision assessment Repeatability intra assay precision (data from linearity assessment on one day) Intermediate Precision- inter assay precision (data from linearity assessment on different days +2 nd operator) Reproducibility precision between laboratories (part of robustness study) Acceptance criteria Repeatability SD<1CT and CV%<5 Intermediate precision - SD<1CT and CV%<5
Precision_2 Results Repeatability results are reported as mean values from data collected from 3 standard curves run on one day Measurement Acceptance value results Standard Deviation Pass/Fail parameter range between data sets Standard Deviation <1.00 Ct 0.30 0.37 PASS CV (%) <5.00 0.876 0.082 PASS Intermediate precision results are reported as the mean values of data generated by operator 1 from all previously analysed standard curves compared with the results from 1 standard curve over 4 days from operator 2. Measurement parameter Acceptance value range Results between data sets Standard Deviation <1.00 Ct Operator 1: 0.30 ± 0.37 Ct Operator 2: 0.25 ± 0.27 Ct CV (%) <5.00 Operator 1: 0.876 ± 0.892 Operator 2: 0.769 ± 0.736 Pass/Fail PASS PASS The RT-qPCR assay fits within the range of acceptable values for Repeatability and Intermediate precision
Parameters for RT-qPCR assay validation Linearity Sensitivity Accuracy Precision Specificity Robustness
Specificity_1 is defined as the ability to distinguish known negative samples from known positive samples Specificity assessment Specificity for Transduced cells as opposed to mock transduced cells Target amplification specificity: - melt curve comparison between control plasmid and tested assays - primer specificity against BLAST Acceptance criteria TCR target present in Transduced cells and absent in mock transduced cells (highest cell number used in the assay) TCR amplified product Tm=84-86 C and ABL amplified product Tm=78-81 C - Single peak only Not binding to endogenous TCR receptor as well as other abundant human gene targets Query cover=100% and E value<<2 (BLAST)
Specificity_2 Results Measurement parameter Acceptance value range Results Pass/Fail Transduced cells Amplified Ct<37 20-25Ct PASS Mock transduced cells Not amplified <37 >40Ct PASS House keeping specific product Measurement parameter Acceptance value range Results Standard deviation between data sets Pass/ Fail TCR plasmid melt peak (plasmid) 84-86 C 84.78 C 0.012 PASS TCR assay melt peak (Standard curve) 84-86 C 85.06 C 0.097 PASS House keeping plasmid melt peak (plasmid) 78-81 C 79.21 C 0.038 PASS House keeping melt peak (Standard curve) 78-81 C 79.71 C 0.068 PASS TCR specific product Measurement parameter Acceptance value range Results Pass/Fail Blast search for House keeping assay Query cover =100% E<<2 Q=100% E=0.0006 Blast search for TCR specific assay Query cover =100% E<<2 Q=85% E=1.4 PASS FAIL Plasmid alignment for TCR specific assay Full alignment of primers sequence with plasmid YES PASS The RT-qPCR assay fits within the range of acceptable values for Specificity
Robustness_1 is a measure of assay capacity to remain unaffected by small, but deliberate variations in method parameters and provides an indication of its reliability Specificity assessment Different laboratory (part of technology transfer) Different operator (2 operators) Different PCR instrument (Quantstudio 6/7 and Quant studio 12K flex) Acceptance criteria R 2 1 +/- 0.1 Slope -3.6 to -3.1 Efficiency 90 to 110% Sensitivity the same across runs
Robustness_2 Results Site 1 Run1 Site 1 Run2 Site 2 Run1 Site 2 Run2 Site 2 Run3 Acceptance Value Range Operator 1 Operator 2 Operator 1 Operator 2 Operator 2 Pass/fail Instrument 1 Instrument 1 Instrument 2 Instrument 2 Instrument 2 Slope -3.6 to -3.1-3.1-3 -3.4-3.49-3.38 R 2 1 +/- 0.1 0.968 0.922 0.99 1 0.99 PASS Efficiency 90-110% 109.7 109.5 96.84 93.43 96.63 Sensitivity The same in all runs 100 100 100 100 100 PASS The RT-qPCR assay fits within the range of acceptable values for Robustness
RT-qPCR assay Validation Summary Measurement parameter Acceptance value range Results Pass/ Fail Linearity Slope -3.6 to -3.1-3.409 PASS Efficiency% 90-110 98.09 PASS R 2 1 +/- 0.1 0.993 PASS Sensitivity TCR specific assay % positive detection of 100 cells (Ct <37) 90 97.5 PASS CV (%) <5 2.87 PASS Sensitivity House keeping gene Precision- repeatability Intermediate Precision Specificity % positive detection of 100 cells (Ct <37) 90 100 PASS CV (%) <5 1.91 PASS Standard Deviation <1.00 Ct 0.37 PASS CV(%) <5.00 0.082 PASS Standard Deviation <1.00 Ct CV (%) <5.00 Operator 1: 0.30 ± 0.37 Ct Operator 2: 0.25 ± 0.27 Ct Operator 1: 0.876 ± 0.892 Operator 2: 0.769 ± 0.736 PASS PASS WT1 melt peak 84-86 C 85.06 PASS ABL melt peak 78-81 C 79.71 PASS Slope -3.6 to -3.1 Values in range PASS Robustness Efficiency% 90-110 Values in range PASS R 2 1 +/- 0.1 Values in range PASS
RT-qPCR controls used throughout Validation Positive Controls - Plasmid vector encoding genetically modified TCR nucleotide sequence - Plasmid vector encoding housekeeping gene nucleotide sequence - 1x10E6 Transduced cells (stock) Negative Controls - NTC no template control (reagent contamination and primers dimers) - NO-RT (gdna amplification) - 1x10e6 Mock Transduced cells (non specific primers binding)
PCR assay workflow to Validation Optimisation (evidence) Qualification (gain confidence and learn about assay limits) Validation (demonstrate that it is suitable for intended purpose!)
RT-qPCR assay Validation Summary
RT-qPCR Validation Summary To validate RT-qPCR assay an understanding of wider assay context is required e.g. patient samples? (GCLP) or manufactured product testing (GMP); Validation has to be specific to the chemistry used (SYBR Green, TaqMan, multiplex PCR); Validation requires evidence of Optimisation and Qualification; Parameters to be assessed during RT-qPCR validation are : Linearity, Sensitivity, (LOD and LOQ not trivial!) Accuracy, Precision, Specificity, Robustness Validation must have PREDEFINED ACCEPTANCE CRITERIA Think carefully about requirement for internal positive controls to control for PCR inhibition e.g. plasmids or synthetic templates;
Other parameters to be considered for PCR assay validations Assay Specificity : Product Sequencing +gel run Assay Specificity: Primer analysis (Primer 3 or other on line tools) Assay sensitivity: RNA/DNA extraction spike control to control for extraction efficiency Validation can take 2-4 months! Think about reagent stability (aliquoting and lot to lot comparison studies) Detection of RNA or DNA? RT efficiency! Consider background of samples that are analysed (is internal positive control required?) MIQE RT-PCR validation analysis = GenEx software
Cell and Gene Therapy Catapult 12th Floor Tower Wing Guy s Hospital Great Maze Pond London SE1 9RT +44(0)20 3728 9500 Cell and Gene Therapy Catapult is a trading name of Cell Therapy Catapult Limited, registered in England and Wales under company number 07964711.