Precision in Quantitative Imaging: Trial Development and Quality Assurance Susanna I Lee MD, PhD Thanks to: Mitchell Schnall, Mark Rosen. Dan Sullivan, Patrick Bossuyt
Imaging Chain: Patient Data Raw data Image reconstruction 123 2346.. 65789 6578.. Image analysis Image processing Data analysis Data output
Clinical Trials: Imaging is an Assay Disease Detection Screening Characterize Disease Diagnosis, eligibility or prognosis Anatomic distribution (e.g. tumor staging) Higher level features (e.g. heterogeneity, vascularity, etc.) Monitor Therapeutic Response Change in features with therapy Anatomic (e.g. RECIST) or functional (e.g. SUV)
Schema Diagnostic accuracy REFERENCE STANDARD OF DISEASE STATE PATHOLOGY CONFIRMATORY TEST FOLLOWUP ENROLLED PARTICIPANT INDEX TEST IMAGING EXAM IMAGE-GUIDED PROCEDURE MANAGEMENT STANDARD OF CARE STUDY DEFINED Biomarker REFERENCE STANDARD OF PATIENT OUTCOME OVERALL SURVIVAL (OS) PROGRESSION FREE SURVIVAL (PFS) OR DISEASE FREE SURVIVAL (DFS) TREATMENT RESPONSE
Cancer Biomarkers Imaging Tissue (e.g. hormone receptors, cytokeratins) PATIENT OUTCOME 1. Overall survival (OS) 2. Progression free survival (PFS) 3. Clinical response Serum (CA125, PSA, AFP, CA19-9) Genetic Genome Expression (e.g. RNA or protein)
What is a good biomarker? Stable technology Available widely Standardized image acquisition Reproducible Range of normal defined Balance state of the art with generalizability Sargent DJ, Rubinstein L, Schwartz L et al. Eur J Cancer 2009; 45: 290.
Variability: Test Retest Same patient, day, scanning protocol but separate imaging sessions Pre-treatment Post-treatment Range of test-retest Conclusion Index test variability precludes detecting pre- vs. post-treatment change. Lankester KJ, Taylor NJ, Stirling JJ et al. Br J Cancer. 2005.93:979.
Signal Requires Data Quality
Precision vs. Bias precise accurate not precise not accurate precise not accurate
Multiple Sclerosis MRI Image acquisition T1 T2 Post-gadolinium T1 Image analysis Number of new or enlarging lesions Number of enhancing lesions
MRI endpoint in MS treatment studies 157 publications from 1995 to 2006
Imaging Chain: Patient Data Raw data Image reconstruction 123 2346.. 65789 6578.. Image analysis Image processing Data analysis Data output
Imaging Manual Hardware and software Scanner calibration Patient preparation Scanning protocol Post-processing Image acquisition manual with a step by step description is part of any prospective study design.
What determines resolution? Physics of acquisition (i.e. modality) Sampling (e.g. matrix, detector size) Filtering and other contributions inherent in the reconstruction
Point Spread Function Patient Image
Image edges approximate anatomy Structure Real Edge Image Edge
Resolution and Sampling 160 x 160 matrix 320 x 224 matrix
Filtering 1 1 1 1 1 1 2 2 2 1 1 2 4 2 1 1 2 2 2 1 1 1 1 1 1
Filtering: MRI Vendor 1 Vendor 2
Filtering: X-ray Vendor 1 Vendor 2
Partial Volume Effects Completely in scan plane Partially in scan plane
Partial Volume Effects: CT Completely in scan plane Partially in scan plane HU = 0 HU = 30-60
Partial Volume Effects: PET lesion lesion Blurred margins Lower intensity
Quantitative Imaging Biomarkers Alliance (QIBA) Started by RSNA 2007 Mission: Improve the value and practicality of quantitative imaging biomarkers by reducing variability across devices, patients and time Build imaging devices that are also measuring devices Industrialize imaging biomarkers https://www.rsna.org/qiba/
QIBA Approach 1. Identify the sources of error and variability 2. Specify potential solutions in the form of profiles 3. Test these solutions 4. Promulgate profiles to vendors and users Purpose of profiles: Advise vendors what must be implemented in their product Communicate the necessary procedures to users
QIBA Profile Activity Diagram Equipment Assessment Subject Preparation Image Acquisition Image Reconstruction Image Analysis Interpretation Manufacturer specification (pre-delivery) Installation specification Maintenance Quality Assurance
ACR Core Lab QA Site qualification Instrument performance Training Monitoring of image acquisition Scan header for protocol compliance On-line technologist qualitative review Periodic radiologist review Centralized image analysis Post-processing Reader study
Require Site Protocol Compliance Type Orientation Pulse Sequence T1 weighted GRE Sagittal Dynamic 3D Field Of View (FOV) 16-18 cm Slice Thickness 64 slices of thickness 2.5 mm Skip Correct Matrix min. 256 x 192 Frequency A/P NEX 2 Phase Wrap NO Fat-Saturation YES Submitted TR Effective TE Scan Duration Flip angle 20 ms 4.5 ms Between 4.5 and 5 minutes <= 45 degrees ACRIN 6657 (I-SPY 1 trial), Nola Hylton, PI
Imaging Chain: Patient Data Raw data Image reconstruction 123 2346.. 65789 6578.. Image analysis Image processing Data analysis Data output
Image analysis: Turning image into data User extracted features Semi automated Automated Feature 1 Feature 2 Feature 3...
Radiomics: Deep Learning Untrained neural network
Radiomics: Deep Learning Trained neural network
Radiomics: Automated Image Analysis Improve diagnostic accuracy
Radiomics: Automated Image Analysis Triage clinical workflow
Semi-automated: Manual Segmentation Large image datasets segmented for tumor Quantitative feature extraction Model predictive indices Integrate with genomic & clinical data for machine learning Aerts HJ et al. Nat Commun. 2014 ;5:4006.
Reader Extracted Features Density Fluid, soft tissue, calcified Shape Round, oval, irregular Size Linear, volume Margin Sharp, blurred, spiculated Intensity high/medium/low/minimal Summary assement BIRADS level
ROC operating points of 108 radiologists reading same mammograms 1.0 0.8 Skill TP 0.6 0.4 0.2 Value judgments 0.0 0.0 0.2 0.4 0.6 0.8 1.0 FP Beam, Layde, Sullivan Arch Intern Med 1996; 156:209-213
Reader Variability: Size progression Increases and decreases of <10% can be a result of inherent variability. Oxnard GR et al. J Clin Oncol. 2011;29:3114-9.
Variability Introduced by ROI Selection Slice 483 Slice 479 Slice 479 SUV=4.0 SUV=5.6 SUV=6.6
Variability Introduced by ROI Selection All ROI protocols show excellent inter-observer agreement (ICC 0.94) Different ROI protocols yield different ADC values Priola AM et al. Eur Radiol. 2016 Aug 11.
Effect of Windowing Soft tissue window Liver window
Effect of Windowing Lung window Mediastinal window Significant measurement differences between window settings (p<0.001). No significant differences in measurement variability between the lung and mediastinal window settings (p>0.05). Kim H et al. PLoS One. 2016;11:e0148853.
Reader Study Reader blinded to reference standard Multiple readers Independent rather than consensus reads Rules for image interpretation Clinical information available to reader Image selection, windowing, order, etc. Choosing index lesions Selecting region of interest (ROI) Definition of positive vs. negative test Washout period between read sessions of paired imaging exams Digital data forms and screen to document reader study A manual defining reader rules and training cases are part of any prospective study design.
Overall trial framework Hypothesis and specific aims Participants Index test Reference standard Data analysis plan (statistics) Conclusions and implications Funding and compliance http://www.equator-network.org/reporting-guidelines/stard/
Index Test (Imaging Exam) STARD 10 - Index test, in sufficient detail to allow replication STARD 12 - Definition of and rationale for test positivity cut-offs or result categories of the index test, distinguishing pre-specified from exploratory STARD 13 - Whether clinical information and reference standard results were available to the performers/readers of the index test STARD 25 - Any adverse events from performing the index test
Steps toward precision: Define image acquisition Equipment, patient preparation, protocol Balance state of the art with generalizability Define image analysis Read rules, training and testing Validate the system Test-retest, reader agreement measurements Build in procedures for ongoing QA