American College Of Radiology Data Science Institute. Keith Dreyer DO, PhD, FACR Chief Science Officer ACR Data Science Institute May 30, 2018

Transcription

1 American College Of Radiology Data Science Institute Keith Dreyer DO, PhD, FACR Chief Science Officer ACR Data Science Institute May 30, 2018

2 ACR Data Science Institute: Participation Advancing data science solutions for radiological care that are clinically relevant, safe, and effective Advisory Board AI Panel Chairs Senior Scientists

3 HEALTHCARE AI CHALLENGES While there are promising publication and initial applications of AI in healthcare There is currently a limited use of AI in clinical care Why? Possible Reasons 1 Clinically effective uses for AI have been poorly defined 2 No standards for clinical integration / care management 3 Large, annotated training sets are difficult to create 4 Currently no successful economic/business models 5 Limitations in current AI/human UX/UI Current Impact

4 Clinical Data Science: Considerations Ethical Legal Commercialization Education Standards Use Cases Economics Content Compute Implementation Regulatory Validation

5 ACTIVITIES SINCE LAST MEETING Create an infrastructure for use case development Build out the education portfolio Engage radiologists, industry and governmental agencies

6 HEALTHCARE AI ECOSYSTEM CLINICAL PRODUCTS NEEDS Mechanisms to integrate and monitor, AI models in clinical practice using real world experience CHALLENGES Standardized methods for AI model validation ALGORITHMS Bring great ideas and clinical needs together Standardized methods to annotate, or aggregate, data for AI model training and testing IDEAS

7 ACR DSI MISSION Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration AI ECOSYSTEM EDUCATION Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms Educate radiology professionals, other physicians and all stakeholders about AI and the ACR s role in data science for the good of our patients

8 EXPANDING THE ACR DSI STAFF New Staff Positions At the DSI Welcome Chris Treml ACR DSI Director of Operations Welcome Alison Loughlin ACR DSI Communications Specialist Now Hiring: - Clinical Informatics Analyst - Data Science Analyst - Computational Math Scientis

9 DSI ACR USE CASE DEVELOPMENT Use Case Development Status All ACR DSI Subspecialty Data Panels have started work - 19 Use Cases in drafting stage - 9 Use Cases in the review stage - 1 Use Case in the experimental stage Examples of use cases under development - Pediatric Bone Age classification - Lisfranc fracture detection and classification - Colon polyp detection - TBI-RADS Industry collaborations

10 VALIDATING AI ALGORITHMS REGULATORY COLLABORATIONS FDA CENTER FOR DEVICES AND RADIOLOGICAL HEALTH Office of Science And Engineering Labs ACR ACR Certify-AI Assess-AI

11 VALIDATING AI ALGORITHMS REGULATORY UPDATE Novel / Future Pathways For FDA Approval Of Software Devices Software as a Medical Device (SaMD) - 21 st Century Cures Act provides guidance of medical device software - Some software applications may not require FDA regulation - FDA is developing guidance for implementation and will that guidance include AI? Medical Device Development Tools Initiative (MDDT) - Recent FDA Guidance August Method, material, or measurement used to assess the effectiveness, safety, or performance of a medical device qualified for use in device evaluation and to support regulatory decision-making within a specified context of use

12 VALIDATING AI ALGORITHMS REGULATORY UPDATE Novel / Future Pathways For FDA Approval Of Software Devices National Evaluation System For Health Technology (NEST) - Intended to shorten the time to market for new technology health care products by developing a system for more robust post-market surveillance - Bring life sustaining, health promoting devices to patients more quickly - Improve the ability to detect safety issues by moving to more active surveillance - Successfully and efficiently harness data from the diverse set of realworld evidence digital information collected from clinical experience in registries and similar tools creating the necessary infrastructure for a national evaluation system for medical devices By leveraging real world data collected as part of routine clinical care, our nation and the public will more fully realize the potential of the digital revolution for the device space

13 ACR DSI MISSION Leverage the value of radiology professionals as AI evolves through the development of appropriate use cases and workflow integration Establish industry relationships by providing credible use cases, help with FDA and other government agencies, and pathways for clinical integration AI ECOSYSTEM EDUCATION Protect patients through leadership roles in the regulatory process with government agencies and verification of algorithms Educate radiology professionals, other physicians and all stakeholders about AI and the ACR s role in data science for the good of our patients

14 ACR DSI EDUCATION UPDATE

15 ACR DSI EDUCATION UPDATE NIH AI Workshop August 2018 NIH / NIBIB AI Research Funding Plan Opportunity to advise the NIH and government agencies Foundational research benefits academic members and practices Translational research potentially supports the efforts of the DSI

16 American College Of Radiology Data Science Institute Keith Dreyer DO, PhD, FACR Chief Science Officer ACR Data Science Institute May 30, 2018

17 ACR Data Science Institute: Participation Advancing data science solutions for radiological care that are clinically relevant, safe, and effective Physicians Technologists Industry Data Scientists Informaticists Health Care Execs Patients Advisory Board AI Panel Chairs Senior Scientists

18 Clinical Data Science: Considerations Ethical Legal Commercialization Education Standards Use Cases Economics Content Compute Implementation Regulatory Validation

19 AI IN DIAGNOSTICS COMPUTED TOMOGRAPHY MAGNETIC RESONANCE POSITRON EMISSION RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY ABDOMINAL IMAGING BREAST IMAGING CARDIAC IMAGING EMERGENCY IMAGING MUSCULOSKELETAL NEURORADIOLOGY NUCLEAR MEDICINE PEDIATRIC IMAGING THORACIC IMAGING INTERVENTIONAL ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY

20 ACR DATA SCIENCE INSTITUTE COMPUTED TOMOGRAPHY MAGNETIC RESONANCE POSITRON EMISSION RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY ABDOMINAL IMAGING BREAST IMAGING CARDIAC IMAGING EMERGENCY IMAGING MUSCULOSKELETAL NEURORADIOLOGY NUCLEAR MEDICINE PEDIATRIC IMAGING THORACIC IMAGING PULMONARY NODULES INTERVENTIONAL LUNG LUNG CANCER SCREENING ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY

21 Lung Nodule Detection Algorithms MODEL 1 Data Acquisition Ground Truth Training Validation Nodule Description 1 Inferencing MODEL 2 Data Acquisition Ground Truth Training Validation Nodule Description 2 MODEL 3 Data Acquisition Ground Truth Training Validation Nodule Description 3

22 MEDICAL IMAGING AI CREATION TO IMPLEMENTATION INTERPRET AI APPLICATIONS X AI CONCEPTS DATA SCIENCE DATA ENGINEERING RADIOLOGY INFORMATION CYCLE AIDEVELOPMENT CYCLE

23 ACR Data Science Institute: Participation Advancing data science solutions for radiological care that are clinically relevant, safe, and effective Physicians Technologists Industry Data Scientists Informaticists Health Care Execs Patients Advisory Board AI Panel Chairs Senior Scientists

24 ACR Data Science Institute: Participation Advancing data science solutions for radiological care that are clinically relevant, safe, and effective Advisory Board AI Panel Chairs Senior Scientists

25 ACR DATA SCIENCE INSTITUTE COMPUTED TOMOGRAPHY MAGNETIC RESONANCE POSITRON EMISSION RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY ABDOMINAL IMAGING BREAST IMAGING CARDIAC IMAGING EMERGENCY IMAGING MUSCULOSKELETAL NEURORADIOLOGY NUCLEAR MEDICINE PEDIATRIC IMAGING THORACIC IMAGING LUNG-RAD INTERVENTIONAL LUNG ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY

26 MEDICAL IMAGING AI CREATION TO IMPLEMENTATION INTERPRET AI APPLICATIONS AI CONCEPTS DATA SCIENCE DATA ENGINEERING RADIOLOGY INFORMATION CYCLE AIDEVELOPMENT CYCLE

27 DSI AI USE CASE - LDCT LUNG CANCER SCREENING HEALTHCARE AI MANUFACTURERS LUNG-RADS TOUCH-AI REGULATORY AGENCIES DSI AI PANELS AI ENABLED REAL WORLD EVIDENCE NEW PATIENT LDCT SCREENING TRAINED AI ALGORITHM DETECTED AND QUANTIFICATION LUNG-RADS SCORING AND RECOMMENDATIONS ACRASSIST-AI RADIOLOGIST INTERPRETATION MEDICAL MANAGEMENT

28 ACR DATA SCIENCE INSTITUTE COMPUTED TOMOGRAPHY MAGNETIC RESONANCE POSITRON EMISSION RADIOGRAPHY ANGIOGRAPHY ULTRASOUND FLUOROSCOPY ABDOMINAL IMAGING BREAST IMAGING CARDIAC IMAGING EMERGENCY IMAGING MUSCULOSKELETAL NEURORADIOLOGY NUCLEAR MEDICINE PEDIATRIC IMAGING THORACIC IMAGING LUNG-RAD INTERVENTIONAL LUNG ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY ANATOMY

29 ACR DSI AI SERVICES AI USE CASE PANELS ACR TOUCH-AI AI APPLICATIONS ACR TOUCH-AI AI CONCEPTS DATA SCIENCE DATA ENGINEERING AIDEVELOPMENT CYCLE

30 ACR DSI AI SERVICES AI USE CASE PANELS ACR TOUCH-AI AI APPLICATIONS ACR TOUCH-AI AI DATA MGMT AI CONCEPTS DATA SCIENCE ACR Author-AI DATA ENGINEERING ACR AUTHOR-AI AI VALIDATION ACR AIDEVELOPMENT CYCLE Certify-AI ACR CERTIFY-AI

31 VALIDATING AI ALGORITHMS Certify-AI Assess algorithm performance specified by use case metrics using embargoed datasets Multiple readers and guidelines for data quality to ensure ground truth consistency Consistent metrics for measuring performance across sites and standards Protect developers intellectual property while ensuring patient privacy and reducing bias

32 ACR DSI AI SERVICES AI USE CASE PANELS ACR AI DEPLOYMENT ACR AI CONCEPTS AI APPLICATIONS DATA ENGINEERING DATA SCIENCE TOUCH-AI ACR TOUCH-AI AI DATA MGMT ACR Author-AI ACR AUTHOR-AI AI VALIDATION ACR Assist-AI ACR ASSIST-AI ACR Assess-AI ACR AI REGISTRY ACR Assess-AI AIDEVELOPMENT CYCLE Certify-AI ACR CERTIFY-AI ACR ASSESS-AI

33 ACR DSI - AI DEPLOYMENT SERVICES For Institutions: Benchmark data For Developers: Algorithm analytics For Regulators: Surveillance metrics

34 ACR DSI AI SERVICES AI USE CASES ACR AI DEPLOYMENT ACR AI CONCEPTS AI APPLICATIONS DATA ENGINEERING DATA SCIENCE TOUCH-AI ACR TOUCH-AI AI DATA MGMT ACR Author-AI ACR AUTHOR-AI AI VALIDATION Assist-AI ACR ASSIST-AI ACR Assess-AI ACR AI REGISTRY ACR ACR AIDEVELOPMENT CYCLE Certify-AI ACR CERTIFY-AI Assess-AI ACR ASSESS-AI

35 MEDICAL IMAGING AI CREATION TO IMPLEMENTATION INTERPRET AI APPLICATIONS CLINICAL CARE IMAGING AI CONCEPTS DATA SCIENCE PROTOCOL DATA ENGINEERING RADIOLOGY INFORMATION CYCLE AI DEVELOPMENT CYCLE

36 ACR Data Science Institute: Participation Advancing data science solutions for radiological care that are clinically relevant, safe, and effective Physicians Technologists Industry Data Scientists Informaticists Health Care Execs Patients Advisory Board AI Panel Chairs Senior Scientists