Precision in Quantitative Imaging: Trial Development and Quality Assurance

Transcription

1 Precision in Quantitative Imaging: Trial Development and Quality Assurance Susanna I Lee MD, PhD Thanks to: Mitchell Schnall, Mark Rosen. Dan Sullivan, Patrick Bossuyt

2 Imaging Chain: Patient Data Raw data Image reconstruction Image analysis Image processing Data analysis Data output

3 Clinical Trials: Imaging is an Assay Disease Detection Screening Characterize Disease Diagnosis, eligibility or prognosis Anatomic distribution (e.g. tumor staging) Higher level features (e.g. heterogeneity, vascularity, etc.) Monitor Therapeutic Response Change in features with therapy Anatomic (e.g. RECIST) or functional (e.g. SUV)

4 Schema Diagnostic accuracy REFERENCE STANDARD OF DISEASE STATE PATHOLOGY CONFIRMATORY TEST FOLLOWUP ENROLLED PARTICIPANT INDEX TEST IMAGING EXAM IMAGE-GUIDED PROCEDURE MANAGEMENT STANDARD OF CARE STUDY DEFINED Biomarker REFERENCE STANDARD OF PATIENT OUTCOME OVERALL SURVIVAL (OS) PROGRESSION FREE SURVIVAL (PFS) OR DISEASE FREE SURVIVAL (DFS) TREATMENT RESPONSE

5 Cancer Biomarkers Imaging Tissue (e.g. hormone receptors, cytokeratins) PATIENT OUTCOME 1. Overall survival (OS) 2. Progression free survival (PFS) 3. Clinical response Serum (CA125, PSA, AFP, CA19-9) Genetic Genome Expression (e.g. RNA or protein)

6 What is a good biomarker? Stable technology Available widely Standardized image acquisition Reproducible Range of normal defined Balance state of the art with generalizability Sargent DJ, Rubinstein L, Schwartz L et al. Eur J Cancer 2009; 45: 290.

7 Variability: Test Retest Same patient, day, scanning protocol but separate imaging sessions Pre-treatment Post-treatment Range of test-retest Conclusion Index test variability precludes detecting pre- vs. post-treatment change. Lankester KJ, Taylor NJ, Stirling JJ et al. Br J Cancer :979.

8 Signal Requires Data Quality

9 Precision vs. Bias precise accurate not precise not accurate precise not accurate

10 Multiple Sclerosis MRI Image acquisition T1 T2 Post-gadolinium T1 Image analysis Number of new or enlarging lesions Number of enhancing lesions

11 MRI endpoint in MS treatment studies 157 publications from 1995 to 2006

12 Imaging Chain: Patient Data Raw data Image reconstruction Image analysis Image processing Data analysis Data output

13 Imaging Manual Hardware and software Scanner calibration Patient preparation Scanning protocol Post-processing Image acquisition manual with a step by step description is part of any prospective study design.

14 What determines resolution? Physics of acquisition (i.e. modality) Sampling (e.g. matrix, detector size) Filtering and other contributions inherent in the reconstruction

15 Point Spread Function Patient Image

16 Image edges approximate anatomy Structure Real Edge Image Edge

17 Resolution and Sampling 160 x 160 matrix 320 x 224 matrix

18 Filtering

19 Filtering: MRI Vendor 1 Vendor 2

20 Filtering: X-ray Vendor 1 Vendor 2

21 Partial Volume Effects Completely in scan plane Partially in scan plane

22 Partial Volume Effects: CT Completely in scan plane Partially in scan plane HU = 0 HU = 30-60

23 Partial Volume Effects: PET lesion lesion Blurred margins Lower intensity

24

25 Quantitative Imaging Biomarkers Alliance (QIBA) Started by RSNA 2007 Mission: Improve the value and practicality of quantitative imaging biomarkers by reducing variability across devices, patients and time Build imaging devices that are also measuring devices Industrialize imaging biomarkers

26 QIBA Approach 1. Identify the sources of error and variability 2. Specify potential solutions in the form of profiles 3. Test these solutions 4. Promulgate profiles to vendors and users Purpose of profiles: Advise vendors what must be implemented in their product Communicate the necessary procedures to users

27 QIBA Profile Activity Diagram Equipment Assessment Subject Preparation Image Acquisition Image Reconstruction Image Analysis Interpretation Manufacturer specification (pre-delivery) Installation specification Maintenance Quality Assurance

28 ACR Core Lab QA Site qualification Instrument performance Training Monitoring of image acquisition Scan header for protocol compliance On-line technologist qualitative review Periodic radiologist review Centralized image analysis Post-processing Reader study

29

30 Require Site Protocol Compliance Type Orientation Pulse Sequence T1 weighted GRE Sagittal Dynamic 3D Field Of View (FOV) cm Slice Thickness 64 slices of thickness 2.5 mm Skip Correct Matrix min. 256 x 192 Frequency A/P NEX 2 Phase Wrap NO Fat-Saturation YES Submitted TR Effective TE Scan Duration Flip angle 20 ms 4.5 ms Between 4.5 and 5 minutes <= 45 degrees ACRIN 6657 (I-SPY 1 trial), Nola Hylton, PI

31 Imaging Chain: Patient Data Raw data Image reconstruction Image analysis Image processing Data analysis Data output

32 Image analysis: Turning image into data User extracted features Semi automated Automated Feature 1 Feature 2 Feature 3...

33 Radiomics: Deep Learning Untrained neural network

34 Radiomics: Deep Learning Trained neural network

35 Radiomics: Automated Image Analysis Improve diagnostic accuracy

36 Radiomics: Automated Image Analysis Triage clinical workflow

37 Semi-automated: Manual Segmentation Large image datasets segmented for tumor Quantitative feature extraction Model predictive indices Integrate with genomic & clinical data for machine learning Aerts HJ et al. Nat Commun ;5:4006.

38 Reader Extracted Features Density Fluid, soft tissue, calcified Shape Round, oval, irregular Size Linear, volume Margin Sharp, blurred, spiculated Intensity high/medium/low/minimal Summary assement BIRADS level

39 ROC operating points of 108 radiologists reading same mammograms Skill TP Value judgments FP Beam, Layde, Sullivan Arch Intern Med 1996; 156:

40 Reader Variability: Size progression Increases and decreases of <10% can be a result of inherent variability. Oxnard GR et al. J Clin Oncol. 2011;29:

41 Variability Introduced by ROI Selection Slice 483 Slice 479 Slice 479 SUV=4.0 SUV=5.6 SUV=6.6

42 Variability Introduced by ROI Selection All ROI protocols show excellent inter-observer agreement (ICC 0.94) Different ROI protocols yield different ADC values Priola AM et al. Eur Radiol Aug 11.

43 Effect of Windowing Soft tissue window Liver window

44 Effect of Windowing Lung window Mediastinal window Significant measurement differences between window settings (p<0.001). No significant differences in measurement variability between the lung and mediastinal window settings (p>0.05). Kim H et al. PLoS One. 2016;11:e

45 Reader Study Reader blinded to reference standard Multiple readers Independent rather than consensus reads Rules for image interpretation Clinical information available to reader Image selection, windowing, order, etc. Choosing index lesions Selecting region of interest (ROI) Definition of positive vs. negative test Washout period between read sessions of paired imaging exams Digital data forms and screen to document reader study A manual defining reader rules and training cases are part of any prospective study design.

46 Overall trial framework Hypothesis and specific aims Participants Index test Reference standard Data analysis plan (statistics) Conclusions and implications Funding and compliance

47 Index Test (Imaging Exam) STARD 10 - Index test, in sufficient detail to allow replication STARD 12 - Definition of and rationale for test positivity cut-offs or result categories of the index test, distinguishing pre-specified from exploratory STARD 13 - Whether clinical information and reference standard results were available to the performers/readers of the index test STARD 25 - Any adverse events from performing the index test

48 Steps toward precision: Define image acquisition Equipment, patient preparation, protocol Balance state of the art with generalizability Define image analysis Read rules, training and testing Validate the system Test-retest, reader agreement measurements Build in procedures for ongoing QA

49