Accuracy, Precision and Measurement Validation

Transcription

1 Accuracy, Precision and Measurement Validation Ángel Alberich-Bayarri, PhD 1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute 2 QUIBIM Quantitative Imaging Biomarkers in Medicine angel@quibim.com

2 I am Founder & CEO of the spin-off company QUIBIM which provides a service of Imaging Biomarkers Disclosure

3 Learning objectives To understand the concept of accuracy and the importance of providing imaging biomarker results close to reference gold standards and true values To understand the concept of precision and the importance of having low variability in the extraction of imaging biomarkers, in order to ensure usability To handle the potential biases that can appear in the imaging biomarkers measurement process and in its validation

4 Accuracy Precision Technical validation Metrology Protocols Phantom calibration Clinical validation Biases Reference technique calibration Relationship with Clinical Endpoints Outline

5 Accuracy Closeness of a measurement to the true value JCGM 200:2008 International vocabulary of metrology Basic and general concepts and associated terms (VIM) BS ISO : "Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions.", p.1 (1994)

6 Precision Closeness of agreement among a set of results JCGM 200:2008 International vocabulary of metrology Basic and general concepts and associated terms (VIM) BS ISO : "Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions.", p.1 (1994)

7 Accuracy and precision High accuracy, Low precision High precision, Low accuracy JCGM 200:2008 International vocabulary of metrology Basic and general concepts and associated terms (VIM) BS ISO : "Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions.", p.1 (1994)

8 Imaging Biomarkers must be both technically and clinically valid Validation

9 Validation Imaging Biomarkers are considered as a Class IIa Medical Device in Europe, therefore, a CE mark must be obtained in order to be considered for clinical use. FDA 510k FDA 510k

10 TECHNICAL VALIDATION OF IMAGING BIOMARKERS

11 Technical validation For the technical validation, we need to incorporate new fields to our network

12 METROLOGY

13 Technical validation Metrology Defined by the International Bureau of Weights and Measures (BIPM) as "the science of measurement, embracing both experimental and theoretical determinations at any level of uncertainty in any field of science and technology." 3 different fields: Units of measurement Realization of units of measurement in practice Traceability to link measurements to reference standards

14 Technical validation Metrology Sullivan DC, Obuchowski NA, Kessler LG, et al. Metrology Standards for Quantitative Imaging Biomarkers.Radiology Aug 12. Epub ahead of print. doi: /radiol

15 Technical validation Metrology Technical validations should be addressed in Facilities for Experimental Radiology and Imaging Biomarkers The experiments allow to add insight into the variability and utility of the candidates to Imaging Biomarkers Technical validation against phantoms and test cases with known conditions is recommended.

16 Technical validation Protocols International initiatives like QIBA or EIBALL are specifying full pipeline protocols (acquisition to processing) resulting from the committee s QIBA Profile: DCE-MRI Quantification v1.0 (Publicly Reviewed Version) (Citation reference)dce-mri Technical Committee. DCE-MRI Quantification Profile, Quantitative Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 1, QIBA Profile: CT Tumor Volume Change v2.2 (Publicly Reviewed Version) (Citation reference)ct Volumetry Technical Committee. CT Tumor Volume Change Profile, Quantitative Imaging Biomarkers Alliance. Version 2.2. Publicly Reviewed Version. QIBA, August 8, QIBA Profile: FDG-PET/CT as an Imaging Biomarker Measuring Response to Cancer Therapy (Publicly Reviewed Version) (Citation reference)fdg-pet/ct Technical Committee. FDG-PET/CT as an Imaging Biomarker Measuring Response to Cancer Therapy Profile, Quantitative Imaging Biomarkers Alliance. Version Publicly Reviewed Version. QIBA, December 11, UPICT Protocol: FDG-PET/CT UPICT (Publicly Reviewed Version) (Citation reference) FDG-PET/CT Technical Committee. FDG-PET/CT UPICT, Quantitative Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 08, 2014.

17 Technical validation Protocols International initiatives like QIBA or EIBALL are specifying protocols (from acquisition to postprocessing) resulting from the committee s periodically organized. Group of potential uncertainties: Patient preparation details Pulse sequences (MR) Angulation Contrast administration Platform for Postprocessing Operative system 1 of the software for postprocessing 1 Gronenschild et al. The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements.

18 Ten, A et al. Chapter of the European Society of Biomechanics Technical validation Phantoms Example 1: Technical Validation of morphometry measurements applied to synthetic trabecular bone analysis from X-ray, CT or MR.: Trabecular thickness, bone volume to total volume, trabeculae separation.

19 Ten, A et al. Chapter of the European Society of Biomechanics Technical validation Phantoms Example 1: Technical Validation of Bone Volume measurements applied to synthetic trabecular bone analysis from X-ray, CT or MR: Trabecular thickness, bone volume to total volume, trabeculae separation. Bone Volume percentage [%] Truth X-Ray CT MR PFC ,319 16,353 19,116 PFC ,191 27,276 27,861 Pore size [mm] Truth X-Ray CT MR PFC PFC

20 Technical validation Phantoms Example 2: Phantom for substances. Recommended to have a refill phantom Applications: Diffusion Iron Fat T1 mapping T2 mapping Image Quality assurance

21 Technical validation Phantoms Example 3: Phantom for PET/CT multicenter studies

22 Technical validation The imaging biomarker is measured with robustness and quality but Covers an unmet clinical need?

23 CLINICAL VALIDATION OF IMAGING BIOMARKERS

24 Clinical validation Biases The biases can be divided in 2: those related to patient characteristics and those related to changes induced by the disease. Patient-related biases: sex, age, race, laterality, Pathology-specific biases: Complete response in PET/CT but metabolism activity mimicking disease progression. In the proof of principle, the most relevant characteristics and potential bias of the population have to be taken into account

25 Clinical validation Reference technique calibration The appropriate standard of reference must be selected in order to validate the results of the imaging biomarker. Example: liver iron concentration [LIC] R2* =10.8+(0.12 x R2*)

26 Clinical validation Reference technique calibration Nevertheless, sometimes the reference technique can have a similar o even lower accuracy and precision that the biomarker that we are proposing (25% disagreement between pathologists 1 ). Example: blind prostate biopsy as a reference standard to validate MR-derived imaging biomarkers Clinical endpoints, taking into account the evolution of the patient is preferred as the reference standard, in order to derive prognostic imaging biomarkers. 1 Elmore JG, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA Mar 17;313(11): doi: /jama

27 Clinical validation Besides being technically validated, at this step, if there is a relationship with disease that adds value to clinical workflow, the biomarker is clinically validated an ready for regulatory application.

28 Conclusions Imaging Biomarkers development workflow require both Technical and Clinical validations to produce a paradigm shift in the management of the disease. Technical validation is closely related to the Metrology science, a field in which our imaging community is just giving the first steps. Clinical validation through Clinical Endpoints is preferred over pathology. Integration in radiological workflow is crucial for promoting the use of the biomarker, after technically and clinically validated.

29 Team Luis Martí Bonmatí MD, PhD. GIBI PI and QUIBIM Founder Ángel Alberich-Bayarri PhD.GIBI Director and QUIBIM CEO QUIBIM Staff Fabio García Castro - M.Sc Rafa Hernández Navarro - B.Sc David García - M.Sc Encarna Sánchez - M.Sc Katherine Wilisch R. - M.Sc CSO CTO Back-End Development of Imaging Biomarkers Business Development Coordinator and CEO support GIBI2 30 Staff Enrique Ruiz Martínez M.Sc Amadeo Ten Esteve M.Sc Ana Penadés - Adm. Internship Students Belén Fos Guarinos Alfredo Torregrosa Lloret Carlos Moya Claramunt Ana Jiménez Pastor Irene Mayorga Ruiz MS Biomedical Engineering Clinical Trials Coord. Administration

30 Accuracy, Precision and Measurement Validation Ángel Alberich-Bayarri, PhD 1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute 2 QUIBIM Quantitative Imaging Biomarkers in Medicine angel@quibim.com